Evaluation of PUM Netherlands Senior Experts 

2012-2015 

Final report 


An independent evaluation study commissioned by the Netherlands Ministry of Foreign Affairs 


Rotterdam, June 2016 

Nico van der Windt (Erasmus University Rotterdam), team leader and country report Indonesia 
Alexander Otgaar (Erasmus University Rotterdam), deputy team leader and country report Ghana 
Marie Heydenreich (Carnegie Consult B.V.) 

Niek de Jong (Erasmus University Rotterdam), country report Colombia 
Hans Slegtenhorst (Carnegie Consult B.V.) 

Hidde van der Veer (Carnegie Consult B.V.), country report Tanzania 
Jan-Jelle Witte (Erasmus University Rotterdam) 



Acknowledgements 


This evaluation would not have been possible without the support of a great number of people and 
institutions in different countries. Staff of the Netherlands Ministry of Foreign Affairs has been 
supportive with providing us with information we needed. They also helped in contacting the 
embassies and relevant stakeholders. The management and staff of PUM and many volunteers were 
without exception very cooperative in providing us with information about their activities in the PUM 
organisation and about the supported SM Es in the field. We are grateful to the persons in the case 
study countries (Colombia, Ghana, Indonesia and Tanzania), both the staff of the companies and 
institutions visited, the local representatives of PUM and other stakeholders who patiently 
responded to our long list of questions. 

Finally, we thank the reference group for their critical remarks to earlier versions of this report, 
stimulating us to further enhance the analysis, which sometimes resulted in a re-phrasing of 
conclusions. We emphasize that the final report reflects the opinions of the research team only, 
based on the evaluation. 

Rotterdam, June 2016 



Table of Contents 


1 Introduction.1 

1.1 Background.1 

1.2 Evaluation.1 

1.3 Organisation of this report.2 

2 What is PUM?.4 

2.1 PUM Organization.4 

2.2 Funding.7 

2.3 Activities.8 

2.4 PUM Portfolio.10 

3 Relevance and additionality.16 

3.1 Howdoes PUM guarantee the relevance of the programme?.16 

3.2 How does PUM assess the relevance of individual projects?.18 

3.3 What kind of Information does PUM have to ensure the relevance of individual 

interventions?.19 

3.4 What is the existing evidence on the relevance of the subsidised programme in the countries 

and sectors where PUM was active?.20 

3.4.1 The relevance of the SME sector.21 

3.4.2 Poverty focus.22 

3.4.3 Relevance for the supported SMEs.23 

3.4.4 Relevance for different types of companies and countries,.26 

3.4.5 Conclusions on relevance.30 

3.5 How does PUM ensure the additionality of lts projects?.31 

3.6 What is the evidence about the additionality of support by PUM experts?.32 

3.7 Is there a link between the number of countries and their status (LDC, LMIC, HMIC) and the 

relevance of the PUM programme? E.g. does a strenger focus on LDCs lead to a more or to a less 
relevant programme?.37 

4 Effectiveness.38 

4.1 How does PUM monitor the results of the programme?.38 

4.2 Does PUM possesses credible evidence of the effectiveness of individual projects and if so, 

what are the conclusions?.39 

4.3 Is it, given available Information, possible to give an assessment of the effectiveness of the 
current geographical and sectoral spread? Would a different spread be more or less effective? .... 45 

4.4 How does the effectiveness of sector specific programmes relate to the effectiveness of the 

individual interventions?.45 

4.5 Is it possible to enhance the effectiveness of PUM through a stronger collaboration with 

other PSD-programmes?.46 

4.6 How does the demand system work? Does PUM have the right incentives for ensuring the 

effectiveness of this system?.47 

4.7 To what extent do business link interventions result in economie relations?.48 

5 Efficiency.51 

































5.1 Given available information, is it possible to give an assessment of the efficiency of the 

management of the programme?.51 

5.2 Is the current geographicai and sectorai spread efficiënt?.53 

5.3 Are sector specific programmes more cost-effective than individuai interventions?.54 

6 Forward Looking Questions.55 

6.1 Is there scope to strengthen the links between ‘aid and trade’ in the PUM programme, while 

maintaining a development focus?.55 

6.2 Is it possible to interest Dutch business more in contributing to PUM?.55 

6.3 Would a closer collaboration of PUM with other PSD programmes lead to a more cost- 

effective approach?.56 

6.4 Is it possible to make better use of information, generated by PUM, for helping to strengthen 

the business climate in countries where PUM is active?.56 

7 Conclusions and recommendations.58 

7.1 Conclusions.58 

7.2 Recommendations.60 

ANNEXES.61 

Annex 1: PUM’s portfolio.62 

Annex 2: Evaluation Methodology.66 

Annex 3: Persons Interviewed.75 


















1 


Introduction 


1.1 Background 

The Foundation Netherlands Senior Experts that executes PUM^ is a non-profit organization founded 
in 1978 by the Dutch employers' organization NCW (currently part of VNO-NCW). The aim of PUM is 
to "promote entrepreneurship and sustainable economie growth in developing countries and 
emerging markets."^ It does so by providing "hands-on advice to small and medium sized enterprises 
(SMEs) and related organisations that lack knowledge and can't afford commercial consultants"^ in 
approximately 70 countries. PUM has access to a pool of around 3,000 experienced Dutch senior 
experts, who all work on a voluntary basis. The mission of PUM is formulated as follows: 

"Our senior experts (all volunteers) provide business advice and technical assistance to 
organisations in developing countries and emerging markets, taking away bottlenecks and 
facilitating sustainable pathsfor growth.""* 

From its start, PUM has mainly been funded by the Netherlands Ministry of Foreign Affairs from its 
development aid budget. Today, more than 90% of the PUM's budget is provided by the Directorate- 
General for International Cooperation (DGIS), under the Ministry of Foreign Affairs^. PUM has been 
regularly evaluated, most recently in 2011®. In that same year the Ministry requested a new 
evaluation in 2015, following the evaluation protocol (2011) and evaluation plan (2012). This 
document reports on the results of the current evaluation, which focuses on PUM's activities during 
the period 2012-2015. 

1.2 Evaluation 

According to its Terms of Reference (ToR),^ the purpose of this evaluation is to assess the: 

1. (continuous) relevance of the PUM programme, including its poverty focus as well as its 
relevance for the "aid and trade" agenda; 

2. improvement of PUM's monitoring and evaluation system; 

3. adequacy of available management information for monitoring and evaluation purposes, in order 
to ensure the relevance, efficiency and effectiveness of the programme; 

4. effectiveness of the PUM-programme during the period 2012-2015; and 

5. efficiency of the management and programme itself. 

In line with this purpose, the research questions formulated in the ToR focus on both the 
management and the processes/procedures of PUM, including its monitoring and evaluation system 
as well as on the effectiveness of the activities that support the SMEs in PUM's target countries. In 
addition, the ToR formulate a number (4) of what it calls forward looking questions. The 
methodology followed in this evaluation is described in the Annex 2 to this report. The evaluation 
approach is based upon quantitative and qualitative information. The quantitative information is 


1 In Dutch, PUM is an abbreviation of 'Programma Uitzending Managers'. However, since the programme not only works 
with managers but also with specialists and entrepreneurs, the abbreviation is only used in combination with the addition 
"Netherlands senior experts", (https://www.pum.nl/about-us/organisation) 

^ "A World of experience" (PUM, undated, downloadable at 

https://www.pum.nl/sites/default/files/images/website/3814.1025_z-card_eng_6panel_v5.def.pdf) 

3 Ibid. 

www.pum.nl 

5 "Annual report 2014" (PUM, 2015). 

® "PUM Netherlands senior experts Evaluatie 2006-2009", Ecorys, 2011 

^ See "Evaluation Foundation Netherlands Senior Experts (PUM) 2012-2015", Ministry of Foreign Affairs, 28 August 2015. 
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mainly from PUM's portfolio data, its monitoring system and from PRIME*, the external monitoring 
and evaluation project, which started about two years ago. The qualitative Information is based on 
interviews with PUM staff, experts and other stakeholders in both the Netherlands and in four case 
study countries: Colombia, Ghana, Indonesia and Tanzania. In these countries in total 79 beneficiary 
SMEs were interviewed and 15 Business Support Organisations (BSOs). Table 1.1 presents the 
distribution by country. 


Table 1.1: Number of SMEs and BSOs interviewed in the case study countries 



SMEs 

BSOs 

Total 

Colombia 

9 

2 

11 

Ghana 

20 

4 

24 

Indonesia 

35 

9 

44 

Tanzania 

15 

0 

15 

Total 

79 

15 

94 


1.3 Organisation of this report 

After this introduction the document continues with a brief description of PUM and its activities. The 
details of the PUM portfolio during the evaluation period 2012-2015 are presented in Annex 1 of this 
document. 

The evaluation results are presented following the evaluation questions formulated in the ToR for 
this study and organised by evaluation criteria: chapter 3: Relevance and Additionality, chapter 4: 
Effectiveness and chapter 5: Efficiency. 

Relevance is defined as "contributing to the Ministry's policy objectives, solving gaps in knowledge 
and skills in SMEs in developing countries and thereby contributing to the creation of jobs, especially 
for poor and vulnerable groups" (ToR). This definition refers to the objectives of the Dutch Ministry 
of Foreign Affairs as specified in its policy letters: the eradication of extreme poverty, sustainable and 
inclusive growth across the globe and success for Dutch firms. This implies that the relevance of PUM 
should also be considered in relation to the Dutch "aid and trade" agenda, as it combines support to 
SMEs in developing countries with the promotion of trade between these SMEs and Dutch 
companies, notably through business links. The concept of relevance is being considered in relation 
to the programme's theory of change (the logic behind the programme) and the "results chain", 
developed to assess the performance of PUM (see Annex 2). Additionality refers to "outputs that 
would not have been achieved in the market without the programme" (ToR). In total there are 7 
evaluation questions on relevance and additionality addressed in 7 sub-sections. 

The ToR define Effectiveness as "a measure of the extent to which an aid activity attains its direct 
objectives" and distinguishes: immediate outcomes (increase in SME knowledge), intermediate 
outcomes (changes in business practices), and ultimate outcomes (changes in SME performance). It 
formulate 8 evaluation questions that are discussed in chapter 4. 

Efficiency is a measure that indicates "how economically resources/inputs (funds, expertise, time, 
etc.) are converted to results" (ToR). When applied to PUM, it is about using "the least costly 
resources possible in order to achieve the desired outcomes" (ToR), which also includes an 
assessment of the running costs of PUM. In the present evaluation, efficiency has been assessed at 
the level of the programme, as recommended by the ToR. Yet, the evaluation questions for this 
evaluation criterion also consider efficiency in relation to the sector and geographic spread and the 
composition of the portfolio (sector-specific versus individual projects). This implies that efficiency 
has been assessed at sub-programme levels such as sector, country and type of intervention. The 


* PRIME: Pioneering Real-time Impact Monitoring and Evaluation in small and medium enterprises. The Centre for the 
Promotion of Imports from Developing Countries (CBI), PUM Netherlands Senior Experts, the Agricultural Economics 
Research Institute (LEI Wageningen UR) and the Erasmus School of Economics (ESE) have partnered in PRIME to design a 
credible and real-time system of impact monitoring and evaluation. 
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section on efficiency includes 4 sub-sections (evaluation questions) on efficiency and cost- 
effectiveness. 

Chapter 6 addresses four forward-looking questions. The first is related to the aid-trade connection 
(section 6.1). The second (section 6.2) focuses on whether the Dutch business community is willing to 
contribute (more) to the PUM programme. The third forward looking question (section 6.3) is about 
cost-effectiveness. The question is inspired by lOB's conclusion that PSD programmes are "often 
implemented in isolation, while recipients often face a broader set of challenges". This section 
therefore addresses the question "Would a closer collaboration of PUM with other PSD programmes 
lead to a more cost-effective approach?". The last forward looking question (Section 6.4) follows up 
on all questions about the quantity and quality of information and data collected by PUM's internal 
and external monitoring and evaluation system (PRIME). 

The document is concluded in Chapter 7 with a summary of the main findings. It also presents some 
recommendations based upon the main conclusions of this evaluation. 

In addition, the Annexes give respectively a detailed description of PUM's portfolio during the 
evaluation period 2012-2015 and of the evaluation methodology. The findings of the four case 
studies - Colombia, Ghana, Indonesia and Tanzania - are reported in separate documents. 
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What is PUM? 


2.1 PUM Organization 

The majority of PUM activities is executed by volunteers. PUM employs around 50 paid staff 
members. As shown in table 2.1, during the evaluation period the number of paid staff increased 
from 48 members in 2012 to 50 in 2013 and 52 in 2014 and 2015. The paid staff is supported by 
approximately 125 staff volunteers. Staff volunteers can be divided into three categories: country 
coördinators, sector coördinators and staff volunteers. They work one to three days a week for PUM. 
The staff volunteers are not paid, but do receive allowances for travel and office appliances. PUM 
experts work on a voluntary basis as well. PUM further collaborates with local representatives, who 
work on a 'no cure no pay' basis. 


Tabie 2.1: Number of staff and staff voiunteers, 2012-2015 (June^ 



2012 

2013 

2014 

2015 

staff 

48 

50 

52 

52 

Volunteer staff 

120 

126 

134 

147 

Country coördinators 

56 

59 

61 

51* 

Sector coördinators 

55 

67 

69 

57 


Source: PUM Database, *June 2015 


Country coördinators 

Country coördinators maintain contacts with the local representatives and are involved in the 
acquisition of new projects. They select and train the local representatives, maintain the local 
networks and visit new or closed projects once or twice a year. They further assess project proposals, 
plan and execute (de-)briefing meetings with the experts who (went) go on missions, and attend 
(internal and external) meetings and netwerking events as a representative of PUM. As shown in 
table 2.1 the number of country coördinators fluctuates between 51 and 61. 

Sector coördinators 

Sector coördinators are particularly responsible for the maintenance of the pool of experts. Their 
tasks entail: assessing the abilities of new expert candidates, conducting intake interviews and 
matching experts with specific assignments. Sector coördinators participate in (de)briefing meetings 
with experts and in evaluation of projects. Furthermore, they organize meetings with all experts to 
discuss policy and professional topics. In 2012 there were 55 sector coördinators. The number went 
up to 69 in 2014 and decreased to 57 in 2015. 

Staff voiunteers 

Staff volunteers assist with the selection projects and intake interviews, assess the quality of experts 
résumés, are involved in the business link programme, coordinate CSR activities, manage the Hans 
Bankert Fund, assist the marketing and communication department and coach new staff volunteers. 
During the evaluation period new functions were created in the areas of recruitment, CSR and 
monitoring. 


^ Mid-2015 PUM introduced a new computer software system, which differs much from the system used up to that time. 
Therefore PUM advised the evaluation team to concentrate on the Information generated from the old system. Some data 
reported in this document in particular data on the portfolio cover the period 2012-June 2015 
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Experts 

PUM Works with high-level voluntary experts, all of them having over 30 years of experience in their 
specific field of expertise. During the evaluation period PUM had access to a pool of approximately 
3,000 experts. The expert pool is renewed continuously as each year some experts are deregistered 
and new experts are admitted. Both access to a wide variety of expertise and the demand of the 
beneficiaries are of central importance for the selection of experts. If an expert is not sent on a 
mission for three consecutive years or shows an apparent lack of interest in his/her area of expertise, 
he/she is deregistered. Experts that have reached a certain age are automatically deregistered (with 
exceptions). As part of the strategie policy priorities for 2012-2015, it was decided to raise the age 
from 70 to 72 years, reflecting society-wide developments such as overall higher life expectancies 
and the raised retirement age. This policy change partly explains the temporarily higher number of 
3,378 experts in the PUM expert pool in 2012 compared to 3,220 and 3,110 experts in the successive 
years (see table 2.2). In case experts have a specific and scarce expertise, they are invited to stay 
longer. Other strategie priorities for the evaluation period included: 

• More active involvement of experts that do not regularly execute projects by engaging them 
offline (sector meetings) as well as online (forum); 

• Development of tools for experts so that they can more easily submit detailed descriptions of 
their expertise facilitating the matching process; 

• Introduction of a project management approach by having the PUM expert providing the 
beneficiary entrepreneur with a project plan that contains clear implementation guidelines; 

• Training of experts to improve their capabilities in the fields of communication, consultation, 
media approach, sustainable production, languages, etc.; 

• Involvement of certain specialised PUM experts in acquisition activities towards potential 
customers when they are present in the region for a regular PUM mission; 

• Active recruitment of Dutch civil servants and policy makers with international experience to 
become PUM experts; 

• Active recruitment of female professionals to become PUM experts. 

Many of these actions were implemented successfully. As of 2012, sector coördinators are supported 
by volunteers with HR-experience in interviewing and selecting (potential) new experts. The potential 
new experts also have to do a 'self-assessment' on cultural sensitivity, the results of which are 
discussed during intake interviews. Finally, in 2014 a workshop on advisory skills was introduced both 
for new experts as well as registered experts. Since the start of 2015 the intake procedure includes a 
formal assessment of the expert. Focus group discussions with coördinators and experts confirmed 
that the intake process has improved as a result of these actions. The actions to recruit more female 
experts, for example by setting up a radio campaign in September 2012, were not as successful as 
expected, as can be seen in the table below. Main reasons being that women are under-represented 
in management functions of the Dutch business community, and that sectors in which there is high 
demand for expertise are dominated by men in the Netherlands, such as Agriculture and 
Manufacturing. 


Table 2.2: OverView of experts, 2012-2015 



2012 

2013 

2014 

2015 

Experts in database 

3,378 

3,220 

3,110 

2,976 

Percentage female experts 

8.0% 

8.9% 

9.2% 

9.7% 


Source: Annual report to Netherlands Ministry of Foreign Affairs (2012, 2013, 2014, 2015) 


Table 2.3 shows that the majority (67%) of experts goes on mission only once per year. Although the 
overall number of missions declined in 2014, the distribution of missions per expert was not affected. 


5 




Only the percentage of experts sent on three missions per year increased slightly from 7.3% in 2012 
to 8.3% in 2014. 


Table 2.3: Experts by number of missions carried out per year, 2012-2015 



2012 

2013 

2014 

2015* 


abs. 

% 

abs. 

% 

abs. 

% 

abs. 

% 

1 mission 

900 

68.7 

897 

66.9 

861 

67.1 

613 

84.3 

2 missions 

289 

22.1 

319 

23.8 

282 

22.0 

94 

12.9 

3 missions 

95 

7.3 

87 

6.5 

106 

8.3 

13 

1.8 

4 or more missions 

26 

2.0 

38 

2.8 

34 

2.7 

7 

1.0 

Total 

1,310 

100.0 

1,341 

100.0 

1,283 

100.0 

727 

100.0 


Source: PUM database, * preliminary 


Other main policy goals for the 2012-2015 period were: 

• Recruitment of staff in knowledge management, business link management and fundraising and 
relationship management. This has resulted in the appointment of staff and volunteers in these 
areas; 

• Implementation of a new method that measures the effectiveness of projects in a selected 
number of industries and a selected number of companies. Measurement is done by PUM- 
experts who have been especially selected and trained for this purpose. Measurement of 
effectiveness has started as a pilot followed by the assignment of a team of external evaluators 
in the partnership PRIME; 

• Increased usage of digital technology to support core activities, e.g. online advice from PUM- 
experts to customers, and the digital exchange of knowledge amongst PUM-experts. The 
relatively old computerized system PRINS has been replaced by a completely new system called 
PROCUS. 

Locai representatives 

Local representatives are inhabitants of the countries where PUM is active. They maintain contacts 
with local firms, acquire new projects, help potential beneficiaries with project proposals and guide, 
if relevant, the visiting experts. Local representatives are compensated with a Standard allowance of 
Euro 400 for each finished mission, half of the allowance is meant for acquisition of the project and 
the other half for guidance and follow-up. The local representatives receive a similar amount for a 
business link or training. Table 2.4 shows that the number of local representatives remained more or 
less the same during the 2012-2014 period, but increased in 2015. The pool of representatives is 
permanently reviewed with a view to clean the pool for representatives who do not bring in 
sufficiënt leads for missions. 


Tabie 2.4: Number ofiocai representatives, 2012-2015 



2012 

2013 

2014 

2015* 

Local representatives 

233 

223 

233 

294 

Country team meetings 

13 

6 

12 

10 


Source: Annual report to Netherlands Ministry of Foreign Affairs (2012, 2013, 2014, 2015) 


The local representatives play an important role in the applications for support from firms and 
Business Support Organizations (BSOs). With their network and their role in the selection of projects, 
they have the ability to influence overall relevance and additionality of PUM. In the previous 
evaluation, the role of local representatives was highiighted and close monitoring was advised. In 


6 





reaction, one of the policy goals for 2012-2015 was to improve the quality of local representatives. 
Concrete steps to be taken in this regard included: 

• Support to local representatives by providing them with (online) training manuals and acquisition 
tools as well as regular country team meetings (CTMs); 

• Training for local representatives aiming at improving their assessment of the commitment of 
entrepreneurs to make the project a success; 

• Regular feedback sessions between local representatives and country coördinators aiming at 
better monitoring of the acquisition activities of local representatives; 

• Shorter term (3-year) agreements with new local representatives; 

• Additional incentives for local representatives on top of the existing 'no cure no pay'-system. 

PUM succeeded in addressing a large number of these issues, for example, through organising 
several country team meetings with local representatives and country coördinators during the 
evaluation period (see table 2.4). These meetings were aimed at discussing the policy changes for the 
period of 2012-2015, such as the integration of CSR practices in PUM activities. Also, the quality of 
the PUM missions and the role of the local representative in ensuring quality were emphasized. In 
some country team meetings an 'intake' clinic took place for practicing interview techniques and 
approach of potential clients. Furthermore, manuals and sector sheets for local representatives were 
developed and distributed in 2012 and 2013, providing guidelines for the selection of projects. 

Finally, in 2013 the term 'Champion representative' was introduced for local representatives who 
recruit quality projects. A maximum of 10 local representatives can be selected yearly for this title. 
The 'Champion representatives' are rewarded with a certificate and a one-time amount of Euro 500. 
The importance of the local representative is recognised by PUM and therefore the performance of 
them is regularly assessed. In addition, PUM more often starts acquisition of projects with a seminar. 
Nowadays PUM promotes the contacts between expert and dient before the start of a mission in 
order to detail the requirements of the assignment. Yet, still today, the quality of local 
representatives seems to be one of the main bottlenecks for PUM's operations. Focus groups with 
sector- and country coördinators as well as with experts showed that the average quality of local 
representatives is still regarded insufficiënt and the incentive scheme not suitable. 

2.2 Funding 

The majority of PUM's activities is funded by the Ministry of Foreign Affairs (both DGIS-DDE and 
DGIS-DSO). Just before the inception of the evaluation period, the Ministry of Economie Affairs 
withdrew its funding, implying a reduction of PUM's budget of €2.5 million per annum. As a 
consequence, PUM management considers it necessary to search for new sponsors. For the 
continuity of its activities, PUM aims at attracting new sponsors who, complementary to the 
programme by the Ministry of Foreign Affairs, would allow PUM to continue its activities at the 
current level or expand these. Following the guidelines of the Ministry of Finance, the explicit aim 
from the strategie plan 2012-2015 was to develop new business models to attract 25% of the total 
budget from other sources“. Potential funders identified beforehand were: 

• (inter)national donors/grant providers; 

• Partners in the private/corporate sector; 

• Private donors; 

• Beneficiaries who contribute to the costs of the missions, for example through introduction 
of the rule to co-finance follow-up mission; and 


“ The Ministry of Finance asks foundations iike PUM to finance at ieast 25% of their activities by externai funds or own 
capitai. 
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Local national governments, today six, that (co-)finance PUM missions. 


As presented in table 2.5, PUM has succeeded in acquiring funds from other sponsors during the 
evaluation period, but not at the expected pace. 


Table 2.5: Expected and realized percentage of PUM budget of other sponsors than the Ministry of 
Foreign Affairs and the Ministry of Infrastructure and the Environment, 2012 - 2015 



2012 

2013 

2014 

2015 

Expected percentage of other sponsors* 

10% 

15% 

20% 

25% 

Realized percentage of other sponsors** 

0.1% 

1.9% 

8.7% 

9.6% 


Source: *Annual report to Netherlands Ministry of Foreign Affairs (2012, 2013, 2014, 2015; ** Annuai report (2012, 2013, 
2014, 2015)) 


Partnerships 

To achieve its goal to become financially less dependent on one financier, PUM maintains regular 
dialogues with other stakeholders, such as (new) sponsors and business partners, politicians and 
media. It further discusses the continuation of partnerships with other organisations that are active 
in the field of private sector development such as BidNetwork and entering into partnership with 
other volunteer organisations for example those supported by VNO-NCW (Ondernemersklankbord 
and Jong Ondernemen). 

PUM hired specific personnel for business development and has also succeeded to diversify its 
funding base via different partnerships for specific missions. In 2014, a variety of missions were 
funded by the Ministry of Economie Affairs, the Ministry of the Interior and Kingdom Relations, FNV 
Mondiaal/CNV international, Argidius Foundation, IFAD, CAIEP and St.-Liberty. The largest additional 
sponsor is Agridius Foundation, who funded 19 missions in 2013 and 26 in 2014. In total Agridius 
Foundation represented 0.9% of the funding budget in 2013 and 1.3% of the funding budget in 2014. 
Other new donors only funded a limited number of missions. 

Forming partnerships and mobilizing funds from the business community proved more difficult than 
expected. The aim was to find one or two firms per sector as a partner. One contract with the bakery 
ingredients company Zeelandia was signed in 2012, however PUM feit that it was conflicting with its 
independence. As a result, this contract was terminated in 2014 and it was decided to focus on 
(inter)national donors/grant providers instead. Funding from recipiënt firms did increase over the 
evaluation period in line with the change in the pricing policy towards mission recipients. 

2.3 Activities 

Missions 

Both firms (SMEs) and business supporting organizations (BSOs) in eligible developing countries and 
emerging markets can apply for a PUM mission. The missions are (usually) executed by a Dutch 
senior expert and take on average about two weeks. This is excluding the preparation and follow-up. 
Follow-up missions frequently take place when more in-depth advice is needed. Most missions 
involve straightforward business and technical advice. Some missions take place in the form of a 
seminar or a quick scan. At a seminar, multiple firms from a certain sector gather for at least two, at 
most three days to get advice on a certain theme, e.g. a joint (sector) problem. The expert who 
organizes the seminar visits the participating firms prior to the seminar to identify common issues 
(quick scan), and often also after the seminar to provide more in-depth advice. Participants of 
seminars are usually representatives (owners) of smaller firms. 
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PUM targets SMEs and BSOs that lack knowledge and are unable to afford commercial consultancy. 
The first advisory mission is 'free' for the customer (the customer only provides accommodation, 
meals and transportation to the expert). Initially, a small voluntary contribution was asked for a 
follow-up mission. Since 2014 a (non-voluntary) contribution of on average €700 is requested from 
the recipiënt firm. The exact amount depends on the development status of the country where the 
follow-up mission takes place as well as on the financial capacity of the firm. Recently a pilot started 
in Indonesia where recipients commit themselves to two missions and pay the required €700 up- 
front. 

Between 2,000 and 2,200 missions per year were planned during the evaluation period. Funding 
from the Ministry of Foreign Affairs is based on an expected annual increase in missions. As can been 
seen in table 2.6 the number of missions remained more or less the same during the evaluation 
period with between 1,875 and 1,964. In 2014, fewer missions took place than expected. The change 
in pricing policy most likely explains part of the decline, but in particular the withdrawing of the 
Ministry of Economie Affairs as funder was important as well. Furthermore, the special programme 
for missions to vocational organizations, VEFIICLE, was phased out. Also, environmental disasters and 
political unrest forced PUM to (temporarily) downsize its activities in a couple of countries. 

Business links 

After a SME or BSO has received advice from a PUM expert, further support is possible by means of a 
follow-up mission, or through the organization of a business link or internship in the Netherlands. 
Contacts with Dutch businesses with the aim to create trade are called a business link. Examples of 
trade in this context are import, export, creating a joint venture, buying (second-hand) capita! goods, 
etc. Trainings in the Netherlands are organised as well for one or two representatives of the advised 
firms, often to familiarize them with the technologies of the Dutch counterpart firm. 

The trade promotion objective of the business link programme is in line with the Aid & Trade agenda 
of the Ministry of Foreign Affairs and as such expanding the business link programme was one of the 
objectives for the evaluation period. PUM planned to conduct about 200 business links missions a 
year. As shown in table 2.6, this target was not reached in 2014 and 2015. In 2012 and 2013 the 
target was met with 220 and 219 business links, respectively. 


Table 2.6: OverView of PUM activities 2012-2015 



2012 

2013 

2014 

2015 

Countries* 

68 

73 

71 

66 

Missions 

1,875 

1,964 

1,900 

1,829 

Business links 

220 

219 

187 

119 


*) Countries with at least one active mission during that year 
Source: Annual reports PUM (2012, 2013, 2014, 2015) 


Hans Blankert Fonds 

An expert might conclude that the firm he/she is advising would make considerable progress with a 
small Capital injection. In such a case, the expert can apply for a contribution from the Flans Blankert 
Fonds, PUM's fund for small projects. (According to the PUM's website the average contribution 
amounts to Euro 2,500.) The fund is usually used for the acquisition of capita! goods, e.g. second- 
hand machines in good condition. The annual overall budget of the Flans Blankert Fund is Euro 
250,000. The majority of the Flans Blankert Fonds budget is donated by the Ministry of Foreign 
Affairs. The past couple of years, increasing amounts are contributed by private funds, for example 
by the Entrepreneurs Foundation. 
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2.4 PUM Portfolio 


Geographic distribution of activities: continents and countries 

The list of countries in which PUM is active complies to a large extent with the OECD/DAC list of 
developing countries and the PSD/DGGF^^ list of countries as specified by the Netherlands Ministry of 
Foreign Affairs^^. Further criteria forthe selection of countries include in practice: 

• The possibility of expediently conducting a PUM programme; 

• Trade potential for the Netherlands, especially in transition countries; 

• Geo-political considerations; 

• Possible synergies with local partners in development cooperation; 

• Safety of the country. 

While the Ministry of Foreign Affairs prefers a greater concentration on a limited number of regions 
and countries, PUM aims to be active in a large and broad variety of countries, motivated by the 
following reasons: (i) PUM wants to fulfill the demand in broad areas of expertise, (ii) the demand 
per country is limited, (iii) PUM wants to have sufficiënt capacity to respond to opportunities, and (iv) 
it allows PUM to show flexibility in times of conflict. Internal research of PUM suggests that the 
programme should operate in about 70 countries in order to be (cost-)efficient. 

Next to the 'hard' criteria described above, the country selection is also based on past experiences in 
the eligible countries. The number of countries in which PUM was active during the evaluation period 
fluctuated: 68 in 2012, 73 in 2013, 71 in 2014 and 66 in 2015. Most of these countries were on the 
PSD/DGGF list of partner countries of the Netherlands. It is PUM's aim to not only direct its activities 
at the Capital of each country, but also on rural areas (e.g. cooperatives). 

The figure below gives some insight into the geographic distribution of PUM's activities (for the 
precise counts and percentages see table Al in Annex 1). The majority of the missions (up to 40%) go 
to African countries, followed by Asia (around 30%), Latin America (just below 20%) and (southeast) 
Europe (about 15%). The share of African countries gradually increased, while the share of missions 
to Asian countries declined somewhat. The share of missions to Latin America experienced a 
relatively strong increase, while Europe shows the opposite trend. 

Changes in geographic distribution are guided by the criteria explained above. At the inception of the 
evaluation period PUM was active in some countries that were not on the PSD/DGGF list of partner 
countries of the Netherlands. It was agreed that activities in these countries would either be 
dismantled or funded from other sources. It was agreed that not more than 15% of PUM activities 
should be funded with funds from the Ministry of Foreign Affairs in countries not on the country list 
and that this will gradually be reduced to nil. During the evaluation period, PUM also initiated 
activities in "new" countries in response to their addition to the country list. Furthermore, the 
decrease in activities in European countries can be attributed to the withdrawal of funding by the 
Ministry of Economie Affairs. 

Indonesia, Vietnam and Nepal are the most "visited" Asian countries. Indonesia received more 
missions than any other country in PUM's portfolio, although the number of missions declined from 
149 in 2012 to 117 in 2014. This decrease was related to the presidential election in this year; in 2015 
the number of missions to this country increased to 159. Vietnam and Nepal, on the contrary. 


PSD: Private Sector Development; DGGF: Dutch Good Growth Fund 

See for example "Wat de wereld verdient: Een nieuwe agenda voor hulp, handel en investeringen", Letter of the Minister 
for Foreign Trade and Development Cooperation to the Dutch Parliament, The Flague, 5 April 2013. 
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experienced an increase of missions: from 61 to 75 and 43 to 67 respectively. In Africa, the top-4 
countries are: Morocco (60-58), Ghana (63-56^^), South Africa (61-61), and Tanzania (42-56). Bolivia 
(63-51), Colombia (51-61) and Peru (47-63) are the most important Latin American countries in the 
programme. The top-3 of European countries is made up of Armenia (47-48), Kosovo (35-34) and 
Bosnia and Herzegovina (20-31). 


Figure 2.1: missions by continent as % oftotai, 2012-2015 (June) 



Between 2012 and 2015, PUM has been most active in lower middle income countries, although its 
share went down from 50% of the missions in 2012 to 43% in 2014 (see figure 2.2 below, and table 
A2 in annex 1). PUM was also very active in least developed countries with an increase in share of 
missions from 24% in 2012 to 33%in 2014. The share of mission to upper middle income countries 
remained more or less at the same level at between 23% and 24%. 


Figure 2.2: Missions by type of country (OESO-DAC categories) as % oftotai, 2012-2015 (June) 


0 % 



■ Least Developed 

■ Other Low Income 

■ Lower Middle Income 

■ Upper Middle Income 

■ other 


Missions by type of beneficiary 

PUM missions are focused on firms and BSOs. During the evaluation period, the majority of the PUM 
missions - more than 80%-was targeted at individual SMEs (see table A3 in Annex 1). Only 4 of the 
5,426 missions to firms are categorized as "missions to big companies", while 9 missions are listed as 
"missions to transformation companies". One of the goals during the evaluation period was to 
increase the number of missions to female entrepreneurs. In light of this aim, a number of MoUs 


Number of missions in 2012 compared with the number of missions in 2014. 
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have been signed with several associations of female entrepreneurs. As a result, the percentage of 
missions to female entrepreneurs increased slightly from 18% in 2012 to 22% in 2014. 

The second type of beneficiaries are BSOs, which are organizations that contribute to the creation of 
an enabling environment for the private sector. In total, PUM realised more than 1,000 missions to 
Business Support Organisations (BSOs), half of which targeted institutions of vocational education. 
These missions are (partly) funded by the VEHICLE programme. As this programme was being phased 
out, the number of missions to these institutions declined substantially from 163 in 2012 to 120 in 
2014. Infrastructural organizations and research and/or development organizations were the second 
and third largest BSO category, however, with far less missions per year (between 30 and 52). 

In 2013 PUM started a pilot for a third type of beneficiary called "PUM Overheid". This pilot was 
executed in collaboration with five Dutch ministries and RVO. Embassies can apply for a short mission 
by an experienced former civil servant in order to provide its local governmental counterpart advice 
on a certain issue. The Embassies were informed of the pilot at the end of 2013 and the first 5 
missions, selected from 27 requests, took place in 2014. Since the Embassies needed more time to 
get familiarized with the pilot it was extended to 2015. 

Sectors and clusters 

During the evaluation period, the policy of the Ministry of Foreign Affairs was focused on four broad 
themes, namely: water (quality and management), food security (from seed to the consumer's plate), 
security and law and order, and sexual and reproductive heaith care and rights. Especially the first 
two themes fit well with PUMs expertise. For all developing (and developed) countries and emerging 
markets, Dutch international economie policy is aimed at nine top sectors. Within these sectors, the 
focus is on knowledge development, innovation, export support and economie diplomacy. PUM is a 
demand-driven organization and its sector focus thus largely depends on mission requests from SMEs 
and BSOs. Nevertheless, PUM did try to direct activities towards the selected top sectors by 
instructing country coördinators to select preferred sectors or focus sectors in their respective 
countries, informing sector coördinators and by selecting new experts in these areas. Also, the 
degree of poverty and as such possible impact in certain sectors is taken into account when possible. 

The overall portfolio of PUM is much broader than the focus sectors formulated by the Ministry of 
Foreign Affairs. During the evaluation period PUM was active in about 70 sectors (the precise number 
depends on the definition of sector), which are categorized in 20 'clusters'^"*. Table A4 in annex 1 
gives an overview of the number of missions to SMEs per cluster. The tables show the very broad 
coverage of PUM activities over multiple sectors. 

Between 2012 and 2015, most missions took place in the 'Tourism & Flotels & Catering' cluster. The 
number of missions in this cluster increased noteworthy too, from 162 missions in 2012 to 233 
missions in 2014. 'Food & Beverages Production' is the second most important cluster for PUM. In 
the clusters 'Stockbreeding & Fisheries' and 'Business Support & Management' a significant number 
of missions took place as well. The most notable decline in missions can be observed for the cluster 
'Metal Industry'. In 2012, 120 missions took place, which declined to 73 missions in 2014. 

Missions by goal 

The PUM mission database gives Information about the goal of the missions, ranging from technical 
support and education and training, to support in the area of marketing, general management, 
export and financial planning. The mission goal 'seminar' is less intuitive, but is by far the least 
common one. Table A5 in annex 1 shows the number of missions that are associated with each of the 
mission goals recorded in the PUM mission database. It is important to note that missions can have 
more than one goal, with two or three goals associated with the same mission being quite common. 


With the transfer to a new ICT system (ProCus) some sectors were combined and a few others were created. From mid- 
2015 onwards there are 71 sectors. 
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The most common mission goal during the period of study was technical support, followed by 
general management. Education and training, and marketing were mission goals with about equal 
frequencies. Missions that help beneficiaries to develop financial plans and business plans were less 
common, while export and 'seminar'are rarely identified as mission goals. 

Figure 2.3 shows the percentage of missions that are associated with these mission goals. For 
example a figure of 60% means that 60% of all missions included 'technical support' as mission goal. 
In many cases these missions also had other goals. 

The figure shows that throughout the research period almost two thirds of all missions were 
associated with technical support, and almost 40% with general management. Education and 
training, and marketing are both associated with about a quarter of missions throughout 2012-2015, 
and financial and business planning with a bit over 10%. These shares are relatively stable over time, 
and apparent changes in 2015 should be interpreted carefully as these data are still preliminary. 
Possibly the goals 'general management' and 'education and training' show a slightly upward trend. 

Figure 2.3: Missions tofirms by goais as % oftotai, 2012-2015 (June) 
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Technical General Education Marketing Financial / Export Seminar 
management and Training businessplan 

Source: PUM Database 

Corporate sociai responsibiiity 

Over the evaluation period, PUM has worked on incorporating more corporate sociai responsibiiity 
(CSR) principles in its activities, especially in transferring such principles to the advised firms. PUM 
focuses on the OECD themes sustainability, environment, labour conditions and safety. In line with 
this policy, a CSR manual for experts was introduced in 2012. Experts are further asked to keep an 
eye on unethical operations and a new debriefing form was introduced with questions on CSR. In 
2014, a number of experts were trained as CSR coaches. Nowadays, each cluster has a CSR coach 
who can be consulted by both experts and sector coördinators when dealing with CSR issues. PUM 
itself introduced a project in 2012 to compensate for its CO^ emissions. PUM reports that it operates 
CO^ neutrally from 2015 onwards. 

Firm Size 

An important selection criterion for PUM beneficiaries is the size of the firm measured in number of 
employees. PUM focuses its support at small and medium sized enterprises. Depending on the 
country and sector, this usually relates to firms with between 10 and 250 employees. In case it is 
expected that support to a larger company would result in large employment effects, the 
management of PUM can decide to support a larger firm. The Annex (table A6) and figure 2.4 give an 


Transfer of knowledge to more than one company during a mission is often labelled as 'seminar'; this is often combined 
with individual support to the participants. 
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OverView of the number of missions by the size of beneficiaries. These figures only include firms, 
since employee counts and sector may have a different meaning when applied to BSOs or 
government institutions. 

The data shows that the majority of PUM's clients is small in size, with 45% having less than 25 
employees and 85% having up to 100 employees. While the group of companies with 100 to 500 
employees is still substantial (approximately 14%) the number of companies above 500 employees is 
insignificant (about 1%). The distribution of missions by firm size is relatively stable throughout the 
period of study, possibly with a slight shift towards smaller firms. 


Figure 2.4: Mission byfirm size (tt of employees) as percentage oftotal, 2012-2015 (June) 
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■ Unknown 


Source: PUM Database 

Follow-up missions 

PUM beneficiaries are allowed to apply for follow-up missions after the (successful) completion of 
earlier missions. As shown in figure 2.5 and table A7 in annex 1, such follow-up missions make up a 
substantial share of the PUM portfolio. The majority of missions are aimed at new beneficiaries, who 
have not received a PUM mission before. But throughout the evaluation period a substantial number 
of missions are follow-up missions to beneficiaries who have participated in PUM before. The share 
of follow-up missions started at 31.3% in 2012, but dropped to about a quarter in later years. The 
decline is related to the introduction of a contribution by the beneficiary to the costs of the follow-up 
mission. Most follow-up missions are the first follow-up visit, but 2"'*, 3'''^, and later follow-up missions 
also occur frequently. 

Missions to BSOs and missions to SMEs differ strongly in their share of new versus follow-up 
missions. While well over half of BSO missions are follow-up missions, missions to firms are much less 
likely to be follow-up missions. Moreover for missions to firms a clear downward trend is visible in 
the share of follow-up missions related to the introduction of the co-finance by the beneficiary. 
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Figure 2.5: New versus follow-up missions as % of the total, 2012-2015 (June) 



■ new 

■ Ist followup 

■ 2nd followup 

■ 3rd or later followup 


Source: PUM Database 
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3 Relevance and additionality 

3.1 How does PUM guarantee the relevance of the programme? 

PUM's primary objective is to reduce gaps in knowledge and skills in SMEs in developing and 
transition countries. As such PUM intends to contribute to sustainable and inclusive growth and the 
creation of jobs with a view to eradicate poverty. As a secondary objective, PUM aims to promote 
trade and investment relations between supported SMEs and Dutch businesses, mainly through 
Business Links. Although this combinatiën of objectives fits well with the current 'Aid and Trade' 
agenda of the Netherlands government^®, it must be realised that PUM focuses on aid rather than 
trade. In Chapter 6 we will discuss whether it is feasible and desirable to put more emphasis on trade 
without compromising the programme's relevance for the provision of aid. PUM attempts to ensure 
the relevance of the programme by applying criteria regarding the selection of countries, regions 
within countries, sectors and beneficiaries (firms, BSOs). 

Selection of countries and regions within countries 

PUM's list of targeted countries is regularly updated and follows with some exceptions the list of the 
PSD/DGGF partner countries of the Ministry of Foreign Affairs. The choice of these partner countries 
is motivated in several policy documents of the Ministry and is based on indicators such as poverty, 
level of income, need for knowledge and expertise available in the Netherlands, etc. PUM's country 
selection during the evaluation period was largely based on the PSD/DGGF country list determined in 
2013^^. In the discussions between PUM and the Ministry it has been agreed that activities in 
countries that are not on this list will gradually be reduced. It is allowed that PUM continues 
presence in non-listed countries with funds from other sources. The last revision of PUM's country 
list implied that the countries in Eastern Europe (mostly upper middle income countries) were taken 
from it because they no longer qualified, given their income levels. Apart from the country list, PUM 
uses several other criteria for the selection of countries: the concentration of SMEs, the World Bank's 
ease of doing business index, a safe and secure environment for PUM volunteers and expert, 
information of local chambers of commerce, branche organisations and previous experiences with 
candidate countries. Security issues sometimes urge PUM to temporarily stop activities in (fragile) 
countries such as in Burkina Faso, after the recent terrorist attack (2016). 

PUM also tries to secure the programme's relevance by selecting eligible regions within countries. 
From time to time PUM reconsiders the regions that qualify for support. For example, it was decided 
to work no longer with local representatives in Jakarta (Indonesia), Bangkok (Thailand) and Manilla 
(Philippines), because these country capitals are considered "too developed" for aid from PUM. 

The regional distribution of PUM missions within countries is strongly dependent on the geographic 
distribution of local representatives. Although PUM is a demand driven organisation and therefore 
potential beneficiaries are expected to take the initiative for missions, the local representatives play 
a prominent role in the generation of this demand. They identify potential 'clients' and help them to 
formulate a request for support, which in most cases is rewarded. Regional income levels do not play 
a role in this demand. Local representatives tend to acquire missions to beneficiaries that are located 
relatively close to their residence. There are some exceptions though: during the visit of the 
evaluation team to Colombia, it became clear that the local representative in Bogota was actively 
promoting support to rural SMEs, rather than SMEs located in urban areas. In contrast, in Indonesia a 
certain bias can be observed towards the more developed regions in the country. Local 

See "Wat de wereld verdient: Een nieuwe agenda voor hulp, handel en investeringen", Letter of the Minister for Foreign 
Trade and Deveiopment Cooperation to Parliament, The Hague, 5 Aprii 2013. 

See footnote 14. The iist was revised in the ietter of the Minister to Pariiament of 19 February 2016. 
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representatives are selected carefully, taking into account the potential for PUM support of the 
regions in which they operate. Apart from these exceptions there are no indications that in the 
selection process of local representatives the income position of the country or the region in the 
country did play an important role. Discussions with PUM staff revealed that selection of regions and 
therefore of local representatives is mainly guided by the presence of potential dient SMEs. 

Selection of sectors 

PUM Works with annual country action plans that specify the sectors to be targeted. As much as 
possible PUM follows the sectors prioritized by the Ministry of Foreign Affairs in the context of its aid 
policies. As a result of this policy, PUM experts are active in (semi-)public sectors such as the water 
and heaith sectors in for example Indonesia, or in sectors relevant for food security (agriculture and 
horticulture), but also in sectors that are not listed by the Ministry as priority. It should be realised 
that PUM support is in principle demand driven and that therefore "dient" demand for support 
determines to a large degree the sector distribution of PUM's activities. For that reason, PUM also 
supports SMEs in sectors that do not belong to the priority sectors of the Netherlands government, 
as long as they are expected to contribute to employment and growth. Sector and country 
coördinators, who are in the end responsible for selecting eligible companies, are immediately 
informed about changes in the sectoral focus. Country coördinators are expected to convey this 
information to local representatives, notably during their bi-annual country visits. 

Beneficiaries 

Statistics ^^show that worldwide SMEs are vital for production and particularly employment. This is 
also true for the countries targeted by PUM. As an illustration table 3.1 presents the shares of micro, 
small and medium sized companies in a selection of Asian economies (measured by the number of 
units, jobs and GDP). Shares in GDP vary from 26% in Malaysia - a more developed country - to 60% 
in China and 76.6% in Cambodia. Shares in employment are even higher, up to 99% in both Indonesia 
and Cambodia. These percentages confirm that policies oriented towards micro, small and medium 
sized companies - such as PUM - can play an extremely important role in employment generation 
and therefore in poverty reduction, thus contributing to achieving goals of the Netherlands' aid 
agenda. 


Table 3.1: Micro, small and medium sized companies in some Asian countries, % oftotal 



China 

India 

Indonesia 

Malaysia 

Philippines 

Thailand 

Vietnam 

Pakistan 

Cambodia 

No. of units 

99.7 

95.0 

99.9 

94.4 

99.6 

98.0 

96.8 

n.a. 

n.a. 

Employment 

74.0 

80.0 

99.0 

40.4 

69.1 

55.8 

96.8 

90.0 

99.0 

GDP 

60.0 

40.0 

58.2 

26.0 

32.0 

47.0 

39.0 

40.0 

76.7 


Note: n.a. =data not available 

Sources: Goh (2007), ADB (2009), UN-ESCAP (2009), Tambunan (2009b). 


In practice, PUM focuses mainly on companies with between 10 and 250 employees and with a 
turnover of less than €10 million. Smaller companies are not eligible for single-company missions (in 
view of efficiency and effectiveness, e.g. the impact on job creation), but are occasionally advised by 
PUM through BSOs, seminars and/or multi-company missions (quick scans). In view of the high share 
of micro companies in developing economies, these alternative modalities - often labelled as sector 
or group approach - are crucial for securing PUM's relevance. Lowering the minimum threshold for 
single-company missions is another way to reach smaller companies, but such a decision is regarded 
less desirable from an efficiency point of view. 


See for example: "Small and Medium Enterprises Across the Globe", Ayyagari, Meghana., Thorsten Beek, and Asli 
Demirgüc-Kunt, World Bank Policy Research Working Paper 3127, August 2003, Washington D.C. 
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Trade and investment relations with the Netherlands 

As explained above aid is PUM's primary objective. Therefore potential trade opportunities are not 
taken explicitly into account in the selection of countries, sectors or companies. Nevertheless, with 
its Business Links programme PUM facilitates beneficiary companies with establishing relationships 
with the Netherlands' business community. The experts and local representatives who are informed 
best about the ambitions of the supported companies are instrumental in the selection of the 
companies that qualify for this special Business Links programme. 

3.2 How does PUM assess the relevance of individual projects? 

Whereas the local representatives play a prominent role in the selection of companies, the final 
decision for support is taken in PUM's headquarters. The country coördinators who are located in 
The Hague instruct the local representative about the acceptance criteria. During the bi-annual 
country meetings, these instructions are fine-tuned and the local experiences are shared among the 
local representatives, including their opinions about the relevance of the programme given the 
country context. This procedure intends to create common understanding about the relevance of 
requests for support. 

PUM assesses the relevance of individual projects by checking if beneficiaries meet a number of 
criteria: 

• the company has to be privately owned; 

• local ownership of the company should be at least 50%; 

• the company needs to be operational for at least two years; 

• the company has between 10 and 250 employees; 

• both the annual turnover and the balance sheet total should not exceed € 10 million; 

• the owner/entrepreneur must show commitment to be fully involved during the entire PUM 
project; and 

• the company is not involved in child labour or forced labour. 

Further, the problem(s) for which the PUM expert is invited should be clearly formulated, as well as 
the expertise needed. The applications for support are screened in PUM headquarters by the country 
coördinator (country knowledge) and the sector coördinator on substantive knowledge applying 
these criteria. Together they decide on which expert meets the requirements formulated in the 
application best. In order to apply the criteria consistently, all applications are finally checked by the 
department ANAMON“. Some of the criteria can be checked easily, such as ownership and annual 
turnover. Often the mere fact that an application is submitted is accepted as commitment from the 
owner/entrepreneur and his/her willingness to accept and implement the changes advised by the 
PUM expert. 

After approval and selection of the expert his/her profile is communicated with the applicant and the 
selected expert is briefed on the basis of the information available from the application. Sporadically 
the expert approaches the potential beneficiary for more information before the application is 
formally approved, although this is not allowed by the PUM organisation. After approval, the 
selected expert is briefed by the country and sector coördinator and he/she usually approaches the 
SME to fine-tune the assignment. The approval process should secure the relevance of the individual 
intervention. 

Only a limited number of applications is rejected (less than 5%) because the requests do not meet 
the criteria, indicating that the local representatives are informed well about the criteria applied. 
Much less applications are rejected because the expertise asked for is not available within the pool of 
PUM experts. 


Department: Analysis and Monitoring (in Dutch: Analyse en Monitoring) 
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In a small number of cases, PUM accepted the request for support even if one or more of the criteria 
were not met. Examples are support to publicly-owned water companies or companies that were in a 
process of privatization. The criterion on the age of the company was waived as well when a 'start- 
up' firm with good prospects needed support. These exceptional cases were all discussed with the 
management. 

Although the criteria seem straightforward they are not always sufficiënt. For example, from the 
interviews it appeared that at least two beneficiaries in Indonesia own more companies than the 
supported one and were quite capable to finance the needed advisory services from their own 
pockets. Also in Ghana we identified at least two firms with relatively wealthy owners. Assuming our 
sample to be representative for the population of all PUM missions, it is advisable to consider the 
application of additional criteria regarding the financial strength of the applicant company owner. 

3.3 What kind of Information does PUM have to ensure the relevance of individual 
interventions? 

Ex ante, the main source of Information for individual interventions is the application prepared by 
the SMEs with support of the local representative. In addition to the description of the problems to 
be addressed during the field visit of the PUM expert they also provide more general Information 
about the SME, such as turnover for a couple of years, number of employees, type of output and 
sector, etc. This Information is checked against the criteria described above. For follow-up missions, 
first mission experts share their experiences during the mission with the country coördinator and 
they report about it in the de-briefing sessions and papers. 

Interviews with the country and sector coördinators and with the experts reveal that in the large 
majority of cases the Information was sufficiënt to determine the need of the beneficiary and to 
select the adequate expertise from the expert database for tackling the issues identified. In case of 
doubt the local representative was consulted and sometimes asked to clarify the request further. 
Interviewed experts reported that in exceptional cases upon arrival the problems to be addressed 
were different from what was presented in the application requiring improvisation of the expert, 
which indicates that for these exceptional cases the Information was not always as adequate as 
required. 

The majority of the SMEs interviewed during the evaluation confirmed that the selected experts 
were capable to address the problems mentioned in the application. Figure 3.1 gives an indication to 
what extent the support of the PUM expert contributed to addressing the problems of the SMEs. 
About 90% of the interviewed firms qualified the support provided by the PUM expert as good to 
very good. A number of interviewees (13 of the 79 SMEs interviewed) reported that there were 
problems in communication, mostly related to language problems in Indonesia and Colombia. 
Nevertheless over 80% of the interviewed SMEs rated the inputs of PUM as good to very good, 
answering to the problems they face, which indicates that the assessment of the relevance of the 
individual intervention on the basis of the ex ante Information was done well. 

In order to learn from current practices and to avoid discrepancies between the problems mentioned 
in the application and the actual issues facing the recipiënt, PUM has set up an internal monitoring 
System with a view to continuously check the relevance (and effectiveness) of lts interventions ex 
post. The Information collected during the application phase is stored in a data base allowing PUM to 
analyse lts activities taking into account the main characteristics of the beneficiaries, such as by 
country and development status of the country, turnover and number of employees of the company, 
etc. 
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Figure 3.1: Performance ofPUM expert and PUM organisation 



Support delivered by the PUM Communication between PUM Overall PUM organization 

expert expert and company staff 


■ Poor BNocomment BGood BVeryGood BDon'tknow 
Source: interviews with 79 SMEs in Colombia, Ghana Indonesia and Tanzania 

Further there are de-briefing sessions with the experts and both, the experts and the beneficiaries, 
are asked to report on paper about their experiences during the visits of the experts. This generates a 
weaith of information on the basis of which the selection criteria and assessment procedures can be 
refined if relevant. The impression from the analysis of the reports from experts and beneficiaries is, 
however, that the experts more often report about logistic issues (difficulties) rather than on content 
related topics, and that the beneficiaries usually give positive reactions. Another problem is that this 
information from the de-briefing notes is of a qualitative nature and therefore not easy usable for 
analytical exercises. As in the case of the ex-ante assessment of the requests the focus is in particular 
on to what extent the gaps in knowledge and practices have been addressed rather than also report 
on long-term effects. 

In reaction to discussions about improving the evaluation practices for private sector development 
programmes with and within the Ministry of Foreign Affairs, a consortium of PUM, the Centre for the 
Promotion of Imports from developing countries (CBI), and an external team of Erasmus University 
Rotterdam and the University of Wageningen has been created to implement the so-called PRIME 
programme, which runs during the 2013-2018 period. PRIME is a monitoring and evaluation 
instrument, which follows annual cohorts of PUM beneficiaries with a view to estimate the effects 
and impacts of the PUM support. Since PRIME is executed in cooperation with external evaluators it 
is independent from PUM itself, which is an advantage. The disadvantage of this approach is that it 
adds an additional layer of bureaucratie procedures as perceived by the PUM experts and the 
beneficiaries. On top of PUM's in-house monitoring system it requires extra inputs from PUM experts 
and beneficiaries. PUM intends to integrate PRIME in its in-house monitoring system when it proves 
to be valuable and feasible. More than the data collected through PUM's internal monitoring system 
the PRIME data allows analytical exercises. In combinatiën, the internal monitoring system of PUM 
and PRIME should deliver sufficiënt inputs to ensure the relevance of the individual interventions of 
PUM. 


3.4 What is the existing evidence on the relevance of the subsidised programme in the 
countries and sectors where PUM was active? 

At the programme level the concept of relevance should be seen in relation to the programme's 
theory of change (the logic behind the programme) and the "results chain" we developed to assess 
the performance of PUM (see Annex 2). This shows that PUM's primary focus is on solving gaps in 
knowledge and practices in SMEs and BSOs, with a clear focus on poverty reduction in developing 
countries. Fiere we discuss the relevance of PUM for the SME sector, the countries in which PUM is 
active and its relevance in addressing the gaps in knowledge and practices in the supported 
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companies and BSOs. Information is from the PUM internal database, PRIME, and data collected 
during the field visits to the four case study countries. 


3.4.1 The relevance of the SME sector 

Several studies show that SMEs in countries where PUM is active are confronted with serious 
difficulties that constrain their performances. Already in 2004, the OECD presented at its 2'''^ 
conference of ministers responsible for small and medium-sized enterprises a paper in which it, 
among others, recommended to "strengthen SME capacities to improve their competitiveness in 
domestic, regional and global markets"^®. For Ghana, Asare^^ presents an overview of the following 
"challenges for SMEs" in this country: "lack of credit facility for small and medium enterprises, 
inadequate infrastructure, low managerial skills, low technological levels/upgrading, weak 
institutional and regulatory framework and globalization". Biro Pusat Statistik (BPS) in Indonesia has 
questioned micro, small and medium companies about the main obstacles they experience. Table 3.2 
summarizes the results of this questionnaire by main PUM region in the country. In total over 2 
million micro, small, and medium sized companies of the 2.7 million respondents mentioned that 
they experience serious problems. Although the country context may vary, similar problems are 
mentioned for the other case study countries: Colombia and Tanzania. 


Table 3.2: Constraintsfacing micro, small and medium companies, by PUM region in Indonesia 



Sumatera 

Java & 

Kalimantan 

Eastern 

Indonesia 

Total 

Lack and/or high prices of raw materials 

23% 

24% 

18% 

23% 

Marketing 

22% 

24% 

21% 

23% 

Lack of Capital 

42% 

35% 

45% 

38% 

Transportation / distribution 

2% 

1% 

3% 

2% 

High prices and/or high prices of energy 

1% 

1% 

2% 

2% 

High cost of and/or lack of skilled workers 

3% 

5% 

4% 

4% 

other 

6% 

10% 

6% 

9% 


Most of these problems are related to external obstacles, and belong as such not directly to the 
domain of the PUM programme. However, PUM experts assist to deal indirectly with these external 
problems. For example, they have advised on logistical issues establishing more efficiënt transport 
and distribution Systems for the supported SMEs in Indonesia and Colombia. In other cases they have 
advised on human resource management addressing the issue of worker skills. In Indonesia the 
owner of the supported SME, which tries to raise its exports to Europe, followed the advice of the 
PUM expert sending his employees to English training courses. Access to finance is a critical issue for 
many SMEs. Although PUM experts may assist companies with getting access to funds, this is clearly 
not the main goal of the programme. PUM aims to transfer knowledge and skills to entrepreneurs 
and employees, thus enabling firms to gain easier access to credit. 

In conclusion, the support provided by PUM to the SME sector is relevant. First, because the SME 
sector is extremely important for employment generation and therefore in containing poverty. 
Secondly, a large number of SMEs in the target countries face problems, and the advices of the PUM 


Promoting Entrepreneurship and Innovative SMEs in a Global Economy: Towards a more responsible and Inclusive 
globalisation, OECD, 2"^ conference of ministers responsible for small and medium-sized enterprises (SMEs), Istanbul, 
Turkey, 3-5 June 2004 

Challenges affecting SME's in Ghana, Andy Ohemeng Asare, 2014, School of Management and Economics, University of 
Electronic Science and Technology of China, Chengdu, China. 
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experts are most valuable to cope with the specific problems they face. The relevante of the 
programme could increase if it would also pay more attention to the huge numbers of micro 
companies in the group of countries PUM is serving. It should therefore be considered whether it is 
feasible to also support this category of companies, that after all generate most of the employment 
and that are confronted with similar problems as the SMEs with more than 10 employees. 

3.4.2 Poverty focus 

The relevante of PUM may be assessed by analysing the degree to which the programme is able to 
generate missions to the least developed countries and regions, notably in view of the ultimate 
objective of reducing (extreme) poverty. In absolute numbers, most missions of PUM targeted 
companies in lower middle income countries (LMICs). Nevertheless, the share of LDCs in the total 
portfolio is substantial, with about one third of all missions (see figure 3.2). Another indicator for the 
programme's poverty focus is the number of projects per 1 million inhabitants^^. It gives an 
indication of how relevant PUM has been in contributing to the ultimate objective of reducing 
poverty in a particular country. Using this indicator, we can conclude that PUM made a more 
substantial contribution to the least developed countries, with around 2.5 projects per 1 million 
inhabitants. Lower and upper middle income countries receive a significantly lower number of 
projects per 1 million inhabitants, which may also be explained by the fact that some of the more 
developed regions within these countries are not eligible for PUM support (Indonesia being a good 
example). Within the OECD-DAC categories we observe substantial differences between countries. In 
Africa, for example, we see at one side of the spectrum a relatively large number of projects in 
Gambia, Burundi and Rwanda, all low income countries, but at the same time also a relatively large 
number of projects in Morocco and Tunisia, both middle income countries. A regression between the 
number of PUM missions per 1 million inhabitants and income per capita and/or the human 
development index does not show any significant correlation. 

Figure 3.2: Number of Projects by main country group, 2012-mid-2015 

Number of projects Projects per mln inhabitants 

I I I li. 

TOTAL LDC TOTAL LM IC TOTAL UMIC TOTAL LDC TOTAL LMICTOTAL UMIC Grand Total 

Source: PUM database 

The data allowed a similar analysis in Indonesia and Ghana. The in-country distribution of PUM 
projects has been compared with the regional income distribution. In both cases there was no 
significant relation between the allocation/execution of PUM projects on the one hand and the 
regional income distribution and/or the regional human development indexes on the other hand. 
Obviously other motives than income (poverty) have played a role in the destination of the visits of 
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PUM projects as a percentage of the total number of SMEs per country would have been a better Indicator, but statlstics 
on numbers of SMEs are not avallable in most of the countries. For Indonesia the total number of SMEs per region In the 
manufacturing sector is strongly correlated with the regional distribution of population, suggestion that population can be 
used as proxy for the presence of SMEs. 
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PUM experts. On the other hand, however, there exist indications that the demand for support from 
SMEs and BSOs in backward regions is smaller than from more developed regions. Reasons being 
that in these countries (regions) the formal SME sector is not developed well as a result of the 
relatively poor social, economie and physical infrastructure and that the informal sector, which does 
not qualify for support from PUM, is relatively big. The withdrawal of PUM activities from Papua in 
Indonesia for reasons that the business environment is not conducive for private sector development 
may serve as an example. 

To conclude: PUM is able to reach the least developed countries and regions, but this poverty focus 
could be further strengthened, e.g. by focusing more on LDCs and backward regions in LMICs and 
UMlCs. 

3.4.3 Relevance for the supported SMEs 

The interviews with the SMEs in the case study countries confirm that the performance of SMEs is 
hindered by a great number of external and internal constraints, which are similar in the countries 
visited for this evaluation. The internal constraints vary from technical issues to management and 
marketing. With virtually no exception the interviewed companies mentioned that the PUM support 
was important for coping with these problems in their companies. In most cases the advice on the 
specific topics for which the PUM expert was invited was considered to-the-point and relevant. 
Nevertheless, not all companies were in a position to fully implement the recommendations because 
they needed additional investments for which finance was not available or too expensive, or because 
the environment in which they operate was not conducive for major changes. Below we present our 
findings about to what extent the PUM intervention was relevant for the individual SMEs 

Achievement of mission objectives 

Relevance can be measured by analysing satisfaction with missions as reported by beneficiaries in 
the PUM monitoring & evaluation system. Data reveal that the majority of the missions accomplish 
their objectives, which suggests that gaps in knowledge and practices might have been reduced. 60% 
of the missions achieve all objectives (answering "yes" to the question "did the mission achieve its 
objectives"), while another 30% achieved part of the objectives. Less than 10% of the missions fail to 
meet the goals. There have been no significant changes in these numbers during the evaluation 
period. It should be realised that these indicators of relevance are based on surveys among 
beneficiaries: they might have an interest in reporting good results, e.g. in view of a possible follow- 
up mission. Therefore, the evaluators have tried to verify the results during the country visits, by 
having interviews with beneficiaries and local representatives, and by visiting the firms. The general 
impression is that most missions indeed manage to realise (part of) their objectives. Most 
respondents are positive about the support delivered by the expert, which is in most cases an 
indication that the objectives were (partly) realised. Also in our small sample (n=79), less than 10% of 
the missions was not successful in meeting the direct objectives (transferring knowledge and skills). 

Through multiple regression we have been able to test the influence of several characteristics of the 
mission and the beneficiary on the success rate (our indicator of relevance). The results demonstrate 
that firm age and the type of mission (first or follow-up) have no significant influence on success. In 
contrast, the size of the company (measured in number of employees) does affect the results 
significantly (at a 10% level). Missions to larger firms tend to be more successful than missions to 
smaller firms, although the effect of firm size is limited. The difference between the chance on 
success of a mission to a firm in the 25**^ percentile and of a mission of a firm in the 75**^ percentile 
(ranked by size) is only 1.3%. Another significant determinant of reported success (=relevance) 
relates to the country in which the beneficiary is located. Missions to firms in least developed 
countries (LDCs) and other low-income countries tend to be less successful compared to missions to 
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lower and upper middle income countries (LMICs and UMlCs). This effect is quite substantial: the 
chance that a mission is successful is 6.7% lower in LDCs. 


Tabie 3.3: Probabiiity ofachieving mission goai(s) (boid indicate statisticaiiy significant^ 



Employees 

Firm age 

Low 

income 

Upper 

middle 

income 

First 

mission 

Constant 

adj. R2 

N 

Coëfficiënt 

P-value 

0.0003 

(0.096) 

0.0006 

(0.487) 

-0.0677 

(0.007) 

0.0289 

(0.296) 

-0.0399 

(0.150) 

0.6651 

(0.000) 

0.0072 

1948 


Through interviews with beneficiaries in the four countries visited for this evaluation we have been 
able to gain deeper understanding of how relevant PUM has been for the assisted companies (ex- 
post). The majority is (very) positive about the mission(s) to their organisation and consider the 
advices highiy relevant. Experts bring practical and useful knowledge: on how to improve quality of 
outputs, how to produce in a more efficiënt manner, how to gain access to new markets, how to 
reduce costs, how to better manage the company, how to use new technologies, how to train staff, 
etc. As a result the gaps in both knowledge and skills are effectively addressed. Figure 3.3 shows that 
about 80% of the interviewed entrepreneurs witnessed an improvement in knowledge and skills in 
their companies as a result of the recommendations of the PUM expert. They are less optimistic 
about the spill-overs of the improvement to their colleagues in the sector. It should be noted that the 
differences between the four countries visited are minimal. 


In our sample of 79 companies, only two beneficiaries were very negative about the missions, while 
one company complained about a recent follow-up mission that was not in our sample (all in Ghana). 
However, also in these missions the ex-ante relevance was clear. In all three projects ex-post the 
relevance was rated low due to conflicts between the expert and the company, not necessarily 
because the advices were irrelevant. In each of these cases, the beneficiary has indicated that the 
expert did not meet expectations, while the expert claims that the beneficiary is not sufficiently 
willing to implement his or her advices. In Tanzania two companies did no longer exist but the 
reasons that they stopped operations are unknown. 


Figure: 3.3: Changes in gaps 
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■ Poor «No effect BGood BVerygood BDon'tknow 
Source: interviews with 79 SMEs in Coiombia, Ghana Indonesia and Tanzania 


Another indicator of relevance is the degree to which advices has actually been implemented. Most 
beneficiaries interviewed in the case study countries state that the majority of the expert's advices 
have been implemented (up to 80%, see figure 3.4), which has been checked during the interviews 
by confronting the discussion partner with some of the recommendations. It turns out that many 
advices were actually implemented during the expert's visit while many other actions were 
implemented soon thereafter. 
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Figure 3.4: Implementation of measures suggested by the PUM expert 
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Source: interviews with 79 SMEs in Coiombia, Ghana Indonesia and Tanzania 

Lack of budget is frequently mentioned as a reason why advices were not implemented. Experts 
sometimes advise to invest in new technologies that are apparently too expensive for the company 
in question, considering the limited access to credit. Nevertheless also these more long-term 
recommendations are considered relevant. Some beneficiaries state that the qualities of staff 
members explain why plans have not been implemented. In one project, for example, the 
owner/entrepreneur States that the person responsible for implementing the advices 
underperformed and that it took a long time before he found replacement. In Ghana many 
discussion partners refer to the increasing inflation and problems with electricity supply. In one 
agricultural project it was the weather that strongly affected the mission, but in this case the expert 
could redefine the focus of the mission. The respondents in Colombia judge the recommendations of 
the PUM experts as very relevant. But also in this country the follow-up of the recommendations was 
sometimes difficult, especially if the recommendations involve investments, again given the limited 
access to finance. A complicating factor in a few cases was also the devaluation of the Colombian 
Peso which made the investment plans unfeasible. In Indonesia a project did not benefit from the 
expert's inputs, though relevant and worthwhile, because it continued to face detrimental policy 
regulations as a result of which the company was forced to discontinue activities (dairy products). 
Another example of difficulties in the implementation of the advice is a micro credit institution for 
women in Indonesia. Although the advice of the expert was considered important and relevant for 
the functioning of this micro credit institution, it could not solve the repayment problems of the 
borrowers. The situation in this particular case was worsened because of the negative economie 
developments in Indonesia. These examples show that the difficult context in which the companies 
operate appear to be a complicating factor during the implementation phase in quite a number of 
cases. There are also other factors that prevent following up of the recommendations. In Colombia 
the establishment of a cooperative of mushroom producers was frustrated because two influential 
members left the cooperative and started their own composting plant. The inputs of the PUM expert 
are considered relevant but the project was put on hold because the remaining members did not 
agree about continuation. 

In contradiction to these rather disappointing cases there are many more for which the gaps in 
knowledge and practices were addressed successfully. Training of car mechanics in Ghana, the re- 
design and production of leather bags in Indonesia, setting up of a butchery in Tanzania which now 
employs 20 people, or support and improvement of the production process of machines for making 
empanadas in Colombia are only a few examples of successful projects. If our sample of interviewed 
SMEs is representative, it shows that only a very low percentage of all missions failed in being 
relevant for the beneficiary. 
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3.4.4 Relevance for different types of companies and countries. 


Impact on knowledge 

Relevance can also be measured by analysing changes in knowledge within supported SMEs as 
reported by PRIME. These data suggest that most companies have experienced an increase in 
knowledge during the year in which they were visited by a PUM expert. Moreover respondents 
estimate that PUM has had (some) influence on the increase in knowledge. Whether or not the 
increases in knowledge can actually be attributed to PUM would require a more in-depth analysis, 
but it is fair to state that most beneficiaries find PUM highiy relevant. Results of PRIME are confirmed 
by our country studies in which a large majority (approximately 80%) declared that the changes 
recommended by the PUM expert positively influenced the collective and individual knowledge in 
the company as shown above. There are no significant differences between the four countries that 
were analysed. 


Table 3.4: Change in knowledge in mission year and one year later, all countries (in % oftotal 
number of respondents) 



Increase in 
knowledge 

PUM had (some) 
influence on this 

Efficiënt ways of organizing the production process or service delivery (N=326) 

74 

93 

Ideas about new product & services (N=329) 

70 

89 

Marketing techniques to increase sales of your product or service (N=327) 

63 

84 

Financial management (N=324) 

53 

73 

Leading, planning and organizing the business (N=328) 

71 

87 

Quality requirements of (inter)national buyers (N=325) 

61 

84 

Ways to retain, motivate and train employees (N=325) 

61 

82 

Effects of the business on the environment (N=329) 

57 

83 


Source: PRIME database 


Table 3.4 presents the results of PRIME in more detail, making a distinction between different types 
of knowledge. It shows for example that a relatively low share of all PUM beneficiaries experienced 
increases in knowledge on financial management (though still more than 50%), while the impact of 
PUM on this change in knowledge is estimated relatively low (though still substantial with 73% of all 
respondents). Higher percentages can be observed when it comes to the contribution of PUM to 
closing gaps in knowledge on how to make the production process more efficiënt, on how to run a 
business and on how to introducé new products or services. 

On the basis of this information, a weighted "PUM knowledge impact" indicator has been 
constructed to take into account that the missions are focused on only one or a few areas^^. In 
addition, the indicator reflects to what extent the PUM advices were responsible for the 
improvements in knowledge. Table 3.5 shows the weights given to the increases in knowledge when 
it was induced by the advices of the PUM experts. The weights vary between zero and eight, ranging 
from no increase as a result of the PUM advice to a great improvement in collective and individual 
knowledge of the company mainly influenced by the inputs of the PUM expert. 


Alternatively a Principal Component Analysis could have been used to reduce the number of result categories. However 
no meaningful components could be identified. 
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Table 3.5: Values of weighted PUM knowledge impact indicator 



PUM influence on knowledge change: | 

No 

effect 

Weak 

effect 

Some 

effect 

Strong 

effect 

Very strong 
effect 

Extent of 

knowledge 

change: 

Increase 

0 

1 

3 

5 

7 

Strong increase 

0 

2 

4 

6 

8 


Statistical analysis (multiple regression^"*) provides insight in the influence of various factors on 
reported relevante as defined by increases in knowledge (see table 3.6). First, it shows that missions 
to companies in low-income countries tend to be more relevant than missions to companies in 
middle income countries. This impact is substantial: the model predicts an increase of the knowledge 
impact indicator with approximately 20% for a mission with an average impact. Interestingly, these 
results seem to contradict the conclusions from PUM data on reported success. However, it could be 
argued that the chance on success in middle income countries is higher (the risks are lower, missions 
are less likely to fail), while in lower income countries successful missions more likely result in 
significant impacts on changes on knowledge. 


Tabie 3.6: Change in knowiedge (boid indicate statisticaiiy significant) 



Empl. 

Firm 

age 

Low 

income 

Upper 

middle 

income 

First 

mission 

Initial 

knowledge 

level 

Production 

Constant 

adj. R2 N 

Coefficients 

-0.0034 

0.0142 

0.5866 

0.2146 

-0.0834 

0.8708 

0.2241 

0.1119 

0.0736 401 

P-value 

(0.143) 

(0.157) 

(0.029) 

(0.475) 

(0.770) 

(0.000) 

(0.357) 

(0.850) 



A second conclusion from the statistical analysis is that the initial level of knowledge has a positive 
influence on the absorption of new knowledge through PUM. Comparing the firm on the 25**^ 
percentile and the firm on the 75*^ percentile regarding the level of initial knowledge, the model 
predicts an impressive 45% impact on the knowledge impact indicator (based on the average 
knowledge impact). This finding is somewhat counterintuitive as one would expect that companies 
with a lower initial level of knowledge have more knowledge to gain. On the other hand, the result 
could be explained under the assumption that firms need a certain level of existing knowledge in 
order to absorb new knowledge^^. However, we need to be careful with drawing conclusions on this 
matter in view of methodological issues: since the questions on initial level, knowledge increase and 
the impact of PUM are asked in the same format, there is a risk that respondents tend to answer the 
questions in a similar way. 

Change in Practices 

Similar to the change in knowledge the change in practice is analysed by subject of the missions. 
Table 3.7 gives an overview of the progress in the different areas distinguished. It records an increase 
in all issues of the PUM missions in the post-mission year, though it is less significant than in 
comparison to the change in knowledge. For the companies that reported an improvement the large 
majority attributed this to the influence of PUM, virtually for all issues addressed by the PUM 
mission. The only negative outlier is the progress and PUM's influence on financial management 


All models in this evaluation are based on OLS. Linear probability model are used for explaining binary dependent 
variables. 

For a classic source on this topic see Cohen and Levinthal (1990), Absorptive Capacity: A New Perspective on Learning and 
Innovation, Adm/n/strot/ve Sc/ence Quorter/y, Vol. 35, No. 1, pp. 128-152 
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showing a relatively low score. Similar to above an impact indicator has been constructed which is 
being used in regression analysis. 


Table 3.7: Change in practices in mission year and one year iater, aii countries 



Improvement 
in practices 

PUM had (some) 
influence on this 

Efficiënt organization of the production or service delivery process (N=285) 

71 

88 

Introduction of new products or services (N=279) 

69 

89 

Marketing techniques to increase sales of your product or service (N=271) 

65 

82 

Financial management (N=274) 

60 

69 

Leading, planning and organizing the business (N=288) 

67 

86 

Meeting quality requirements of (inter) national buyers (N=281) 

64 

86 

Retain, motivate and train employees (N=274) 

64 

82 

Managing the effects of the business on the environment (N=274) 

55 

83 


Source: PRIME database 


The PUM results chain suggests that when beneficiaries acquire new knowledge during a PUM 
mission, this will lead them to also improve their business practices and hence improve their 
performances. Table 3.8 clearly confirms this expectation, as the change in knowledge (controlling 
for other beneficiary characteristics) is a significant and strong predictor of change in practices. The 
adjusted shows that almost two-thirds of the improvements in practices is explained by 
knowledge change. As above, the initial level of practices is strongly and positively correlated with 
the extent of practices change. 


Table 3.8: Change Practices Regression for different sets of variables (bold is statistically significant) 



Employees 

Firm 

age 

Low 

income 

Upper 

middle 

income 

First 

mission 

Initial 

practices 

level 

Production 

Change in 
knowledge 

Constant 

R2 

N 

Coëfficiënt 

-0.0009 

0.0015 

0.0663 

0.1701 

0.0432 

0.2928 

-0.1335 

0.7318 

-0.2458 

0.62 

340 

P-value 

(0.571) 

(0.823) 

(0.713) 

(0.392) 

(0.821) 

(0.010) 

(0.414) 

(0.000) 

(0.541) 




Most of these findings were confirmed in the interviews with the beneficiaries during the field visits, 
though admittedly, not all could be checked physically. Virtually all interviewees mentioned that 
PUM had an effect on the changes in both the knowledge and practices to the benefit of the 
company. It, however, became clear from the interviews that "unfortunately" not all 
recommendations could be implemented fully or quickly because of the local constraints, of which 
access to finance was mentioned most often. This might explain the relatively low score on 
improvement in knowledge and practices for financial management. In spite of the changes 
introduced as a result of the inputs of the expert, the difficulties in accessing external finance 
continued to exist. Some interviewed SMEs pointed to this issue. 

The regression analysis of the effect of PUM on knowledge and practices above is based on PRIME 
data about the first available cohorts and the findings during the field visits. A more detailed analysis 
can be executed as soon as PRIME data will come available about performance of later cohorts. The 
limited size of the currently available dataset has at least two consequences. First, any results that 
are found should be understood as having a wide confidence interval, meaning that the actual effects 
may be higher or lower than the estimates reported here. Secondly a small dataset means that 
relationships between variables are more likely to become insignificant, even if using a larger dataset 
would have revealed them to be significant. 

Gender baiance and CSR 

In the evaluation we paid specific attention to changes in practices regarding gender baiance and 
CSR. Many persons interviewed during the field visits found it difficult to attribute changes in the 
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position of women, if there are any, to the inputs of the PUM expert (see figure 3.5). 40% mentioned 
that they don't know. Similar reactions were given on the questions regarding CSR topics. PRIME has 
also asked the beneficiaries about some other variables relevant for this evaluation, such as worker 
safety. These are reflected in Table 3,9. It shows that in the area of worker safety about 25% of the 
surveyed companies did not have a worker safety policy/system by the end of 2014. More than one- 
third of these companies introduced such a system after the visit of the PUM expert half a year later. 


Figure 3.5: Effects on gender and CSR 

80% 



Adoption of CSR practices Adoption of CSR practices Gender balance in our Gender balance in our 
in our organisation in our sector organisation sector 


■ Poor «No effect «Good «Verygood ■ Don't know 
Source: interviews with 79 SMEs in Colombia, Ghana, Indonesia and Tanzania 

Two-thirds of beneficiary firms already had a website by the end of 2014. Only a small share of the 
one-third that did not have a website in 2014 started one by mid-2015 (17%). Promotion material 
shows similar trends, although a higher share of firms without promotional material in 2014 
introduced it the year after (30%). Half of the firms had a customer satisfaction system by end 2014, 
and a quarter of those who did not have such a system has introduced one by mid-2015 
By the end of 2014 43% of the firms had a marketing plan and the same percentage had a quality 
assurance system. 19% of those that lacked the marketing plan and quality assurance system have 
introduced these by mid-2015^®. 


Tabie 3.9: Systems and poiicies for various business activities, end 2014 and mid-2015, aii countries 



Present by end 2014? 

..if not, introduced by mid-2015? 


%yes 

% no 

%yes 

% no 

Worker safety (N=554) 

76 

24 

36 

64 

Website (N=617) 

66 

34 

17 

83 

Promotion material (N=614) 

65 

35 

30 

70 

Customer satisfaction (N=614) 

50 

50 

26 

74 

Marketing plan (N=608) 

43 

57 

19 

81 

Quality assurance (N=592) 

43 

57 

19 

81 

Environment (N=578) 

30 

70 

14 

87 


The large majority (70%) of firms that were surveyed did not have a system in place for monitoring 
the effects of production on the environment by the end of 2014. Only 14% of this group had 
introduced it by mid-2015 after the visit of the PUM expert. Again it is not clear whether this 
introduction can be attributed exclusively to the expert's inputs. The country visits do not clarify this 
issue, either because a large number of interviewees were not able to answer the question about 
influence of the PUM expert on the decision to introducé CSR Systems (see figure 3.9). It should be 
noted that this issue is not relevant for a large share of the interviewed firms, because they do not 


These similar scores seem coincidence rather than a data issue, as the questions are not directly consecutive 
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have a production process (e.g. services firms). However, the result is similar if the data are corrected 
for the companies not active in producing commodities. 


3.4.5 Conclusions on relevance 

Literature and statistics show that SM Es employ most of the labour force in the target countries of 
PUM. Our findings further provide evidence that PUM has been instrumental in solving gaps in 
knowledge and practice in most of the SMEs supported. During the interviews with the beneficiaries 
it was reported that about 20% of the companies faced difficulties in implementing the 
recommendations because of external constraints, of which access to finance was mentioned as the 
most constraining factor. Not because the recommendations were irrelevant. 

Regression analysis shows that missions to larger firms have a higher chance of achieving their 
goal(s), while it also shows that missions to low income countries have a lower chance of achieving 
their goal(s). Our statistical analysis further indicates that there is a strong relation between the 
improvement in practices as a result of PUM's intervention and the changes in knowledge. Missions 
to low income countries on average result in a higher pay-off in terms of new knowledge compared 
to missions to middle income countries. A second result is that the level of knowledge before the 
start of the mission has a strong impact on the extent to which new knowledge is acquired by the 
beneficiary during the mission. Firms seem to need a certain level of existing knowledge in order to 
be able to 'absorb' the new knowledge offered to them through the PUM mission. 
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3.5 How does PUM ensure the additionality of its projects? 

The Donor Committee for Enterprise Development (DCED)^^ suggests to apply a flow diagram for 
checking the additionality of a project, using the following criteria. First, it identifies the following 
three essential criteria, of which at least one should be met by a project to be additional: 

1. The company has insufficiënt funds to self-finance the project (within a reasonable time frame) 

2. The company lacks the knowledge or competencies to design and/or implement a business 
model in a way that maximises poverty-reducing or other development impacts. 

3. Without the public subsidy the company would be unwilling to implement the proposed business 
model and/or changes in operational standards because of a perceived negative balance of 
costs/risks and benefits. 

During the application phase PUM indeed checks on whether the company is able to finance the 
support itself. The companies are requested to inform PUM about their financial performance during 
the preceding two years. This information is cross checked with the opinion of the local 
representatives. During the field visits, the local representatives confirmed that they do consider the 
applicant's ability and willingness to pay for the provided services. 

The mere fact that the company submits a request for support focused on improvement of 
knowledge, skills and practices is considered evidence of the lack of competences to realize the 
required changes itself. 

Even in cases that the company is financially capable to hire outside expertise the applicants hesitate 
to do this because they are uncertain about the potential benefits, which is shown in the application 
and again confirmed by the local representatives. 

DCED further presents three conditions that should all be met: 

4. The company cannot access the services offered by the publicly-funded agency on a commercial 
basis - whether commercial bank funding or advisory support of similar quality. 

5. The cost-shared project does not displace other companies already operating in the market, or 
that are ready to undertake the same project without public support. 

6. The cost-shared contribution does not duplicate other donor-funded support - whether grant, 
in-kind advice, loan or equity. 

PUM itself does not provide the services on a commercial basis, but there might be alternative local 
suppliers that could provide similar services. Ex ante, it appears to be difficult to assess whether such 
expertise is indeed lacking in the countries (or internationally), and whether the SMEs/BSOs are not 
able to pay for these services, if available. It requires a complete overview of what is available in this 
field and at what prices. Some knowledge about service provision in the target countries is available 
within PUM, but generally it is limited. The local representative is usually better informed about the 
relevant local situation and takes this into account in his "sales" of PUM in his country. During the 
field visits, the local representatives also confirmed that they do consider the questions of availability 
of alternative support and the (potential) beneficiary's ability and willingness to pay for these locally 
provided services. This pre-selection might explain that only a limited number of requests is rejected 
because of (potential) unfair competition with local service providers. Three years ago, PUM 
supported the establishment of a local senior expert organisation in El Salvador (Sabios y Expertos). 

In a couple of other countries (Peru, Indonesia and Thailand), similar organisations are being 
prepared with help of PUM. These local expert organisations are better equipped to judge whether 


Demonstrating Additionality in Private Sector Development Initiatives, A Practical Exploration ofGood Practicefor 
Challenge Funds and other Cost-Sharing Mechanisms, Melina Heinrich, The Donor Committee for Enterprise Development 
(DCED), April 2014 
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local expertise is available at affordable prices, which might improve the assessment of whether PUM 
displace existing companies in the local market that are ready to undertake the same project without 
public support. 

The approach of PUM, namely attaching one expert to one SME/BSO for support at a time is rather 
unique in the world of donor support to the private sector. In some countries, e.g. Indonesia, the 
government is offering special low costs training to local SMEs, but mostly on very general topics that 
often are not addressing the specific issues for which PUM experts are invited. For technical issues or 
expertise on international marketing of products, in particular exports to Europe, availability of local 
specialised knowledge is often not available. 

Finally, DCED mentions two extra conditions that are not essential, but indicative of additionality: 

7. Public support leverages investment by other entities that would otherwise not be forthcoming. 

8. Conditions attached to the cost-sharing project, or agency activities complementing it, are 
expected to have a positive influence on wider business operations, the business environment or 
other institutional factors. 

The review of the applications submitted to PUM does not systematically take into account the 
potential synergy with or catalysing role for other PSD programmes. PUM has approached several 
other public institutions active in PSD support with a view to cooperate, which has resulted in joint 
activities but so far not many. The leverage of investments by other entities is not a criterion in the 
approval process of PUM of individual applications. Indirectly PUM support could have a positive 
effect on the possibilities to access external funding from the banking sector. 

Although PUM recognizes that its support to individual companies might have an positive effect on 
other companies in the sector or the region through for example demonstration effects, it is not 
taken into account in the approval process. This is different for support to BSO's and programmes 
that by definition are focused on assisting similar companies in the same sector. The purpose of 
these programmes is exactly that they have a wider influence than on a single company only. 


3.6 What is the evidence about the additionality of support by PUM experts? 

This section discusses the evidence about additionality following the DCED criteria presented above. 

1. The company has insufficiënt funds to self-finance the project (within a reasonable time frame) 

2. The company lacks the knowledge or competencies to design and/or implement a business 
model in a way that maximises poverty-reducing or other development impacts. 

3. Without the public subsidy the company would be unwilling to implement the proposed business 
model and/or changes in operational standards because of a perceived negative balance of 
costs/risks and benefits. 

A proxy for the difficulties SM Es face in financing advisory services is their access to loans from the 
banking sector. Information from PRIME shows that more than 50% of the SMEs responding to 
PRIME's questionnaire had difficulties to access finance from a bank or financial institutions (see 
figure 3.6) for investments. It is more difficult for small companies. This is consistent with the field 
visit reports, Information from the international literature^* and the survey among micro, small and 
medium sized enterprises by Biro Pusat Statistik in Indonesia. The results of the PRIME survey also 
show that SMEs (supported by PUM) in Least Developed Countries (LDCs) experience more 
difficulties with accessing finance. 


See the literature referred to in footnote 5 and 6 
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Figure 3.6: Access tofinance 

By company size (measured by employment level) 
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Another aspect of additionality, besides access to finance, is the potential of beneficiaries to improve 
business practices without support (DCED criterion 2). A useful indicator for this potential is the self- 
reported level of business practices on relevant types of knowledge and skills prior to the mission, as 
registered by PRIME^®. Overall, about 45% of the supported firms report an initial level of practices 
between poor and good: it may be assumed that they meet DCED criterion 2. The other 55% report a 
good to very good level of initial practices: for them it is less likely that they meet this criterion.. 
However, it still possible that they lack the knowledge to further improve their level of practices. We 
also need to take into consideration that beneficiaries tend to give a positive answer to this question 
in order to avoid being perceived as incompetent. There are no clear differences in initial business 
practices by OECD-DAC country status and by firm size 


Figure 3.7: Level of initial practices, average and by company size (N=382) 



Source: PRIME, own elaboration 


The two indicators of additionality (poor access to finance and a low initial level of practices) can be 
combined, by asking what share of beneficiaries face both difficulties. Table 3.10 shows that there is 


This indicator is a weighted average of the initial levels of business practices for types of knowledge and skills explicitly 
addressed by a mission. For example: if a mission focuses on three aspects, each aspect takes a share of 33.3 per cent. 
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no clear relation between OESO-DAC status and the indicator of additionality based on access to 
finance and level of relevant initial practices. 


Table 3.10, Beneficiaries facing poor access to finance and low initial level of practices by country group 



Poor access to finance and low initial level of practices 


no 

yes 

Total 


abs. 

% 

abs. 

% 

abs 

% 

Low Income 

80 

60.6 

52 

39.4 

132 

100.0 

Lower Middle Income 

97 

58.4 

69 

41.6 

166 

100.0 

Upper Middle Income 

55 

65.5 

29 

34.5 

84 

100.0 

Total 

232 

60.6 

151 

39.4 

382 

100.0 


Source: PRIME, own elaboration 


Table 3.11 provides insight in the influence of several variables on the probability that a company 
combines poor access to finance with a low initial level of practices^® (a strong signal of additionality). 


Table 3.11: Probability of reporting difficult finance and low practices (bold indicate statistically 
significant) 



Upper 

First mission Firm age Low income middle Constant 

income 

adj. 

N 

Coëfficiënt 

P-value 

0.1626 -0.0034 -0.0494 -0.0742 0.3397 

0.010 0.100 0.391 0.256 0.000 

0.0172 

382 


As may be expected, when a beneficiary is receiving its first PUM mission it is significantly more likely 
(its probability increases by 16.3%) to belong to the set of beneficiaries with the highest additionality, 
compared to firms receiving a follow-up mission. Moreover younger firms are also more likely to 
belong to this group, although the difference is relatively small and the effect is significant at the 10% 
significance level. Moving from the 25**^ to the 75'*^ percentile of age (from 5 to 18 years) decreases 
the probability of belonging to the firms with highest additionality by 4.5%. Finally, OECD-DAC 
country status of the beneficiary was not a significant predictor as was already shown in table 3.9. 

So: if PUM sends more missions to younger firms and focuses on new beneficiaries rather than 
follow-up missions, its additionality would be expected to increase. Sending more missions to smaller 
firms increases additionality because they are more constrained in access to finance, not in terms of 
targeting beneficiaries with a bigger gap in initial level of practices. 

4. The company cannot access the services offered by the publicly-funded agency on a commercial 
basis - whether commercial bank funding or advisory support of similar quality. 

5. The cost-shared project does not displace other companies already operating in the market, or 
that are ready to undertake the same project without public support. 

6. The cost-shared contribution does not duplicate other donor-funded support - whether grant, 
in-kind advice, loan or equity. 

Ex post PUM collects evidence on the additionality of support through the expert debriefing form. 
This form, however, changed during the evaluation period. For missions in 2012 and 2013, experts 
were asked to indicate: 1) if the supplied knowledge was available locally; 2)if the company would be 
able to contribute 700 euro (international travel costs); and 3) whether or not the applicant received 
support from other organizations. In missions after 2013/2014, experts were asked to discuss the 


Using a limited probability model based on OLS. Firm size (number of employees) and whether a firm is active in the 
production of goods or not were also tested, but were found to be insignificant. 
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ability to pay for a follow-up mission and to report the findings in a more qualitative way. Therefore 
it is not possible to analyse the reported additionality of all missions in the portfolio. For the visited 
countries we have analysed additionality in a more qualitative manner by comparing the feedback of 
the expert with answers of beneficiaries during the interviews. 

In Colombia the principal reason to apply for PUM is to get access to international best practice 
knowledge and management or production skills. Most beneficiaries in this country stated that there 
was no alternative to PUM offered by local consultants. Several of them would have been prepared 
to pay something for the (first) mission, had they known in advance the benefits they received from 
it. For some, the cost would be prohibitive, especially given the devaluation of the currency last year. 
Some of the beneficiaries mentioned 'trust' as an advantage of the PUM programme, as the experts 
do not have any personal interest or commercial motivation at all in the relation with the enterprise. 
The same holds for the selected BSOs, with the exception of one BSO that had received assistance 
from other organizations. Flowever, this BSO also reported that there were no local commercial 
consultants with similar expertise and it was not able to pay for a local commercial consultant (if such 
a consultant would have been be available). 

In Ghana nearly all beneficiaries indicate that local knowledge was not available or affordable. In 
general local consultants do not have the required knowledge, and if they do, they are too expensive 
or not willing to offer the same as PUM does: an intensive two-week training programme 
(consultants often work on several projects at the same time). Discussion partners emphasized that 
PUM offers a distinctive product: concrete, hands-on advice that can be implemented straight away. 
In their view local consultants are often "good in writing reports, not in helping firms to actually 
change". 

Some regional differences can be observed in the country. The ability to pay tends to be higher in 
Greater Accra than in other regions. Beneficiaries in these regions seem to be more aware of 
alternative Solutions for their problems, including local consultants and international business 
support programmes. While for several firms outside the capita! region, PUM is often the only 
available and affordable alternative. Accra-based firms seem to make a weigh-off between costs and 
benefits. For them, PUM is a relatively affordable way of getting access to relevant knowledge. Many 
beneficiaries - also outside Accra - state that they are willing to pay for PUM, if advice leads to 
results. They are, in general, not willing and able to pay the full amount of 6,000 euro (the estimated 
value of a mission). A contribution between 250 and 750 euro (partly covering the international 
costs) seems more reasonable, at least for the companies that have some budget available. 

In Indonesia it was mentioned that domestic expertise on more general topics, such as general 
management, human resource management, and domestic marketing could indeed be provided by 
local professional service providers. But at costs that are much higher than the costs related to the 
visits of the PUM experts. In most cases the SM Es reported that they could not afford hiring local 
professionals and therefore would not have invited them for advice. As in the case of Colombia for 
technical issues or expertise on international marketing of products, in particular exports to Europe, 
availability of local expertise is not available. Another issue mentioned is that local experts try to 
maximize their revenues through also claiming involvement in the implementation of the advice. 
Virtually without exception the interviewed beneficiaries indicated that the required expertise was 
not available, and if available (also for example internationally) at costs that the recipiënt SMEs could 
not afford. Many of the interviewed entrepreneurs further mentioned that they were in particular 
interested in the knowledge of experienced Dutch colleagues. 

For Tanzania the smaller companies visited report that they would have been unable to cover the 
costs and also would find the costs for a follow up mission prohibitive. Companies with 25-100 
employees sometimes would have been prepared to pay more than accommodation and food but 
only after they had experienced the usefulness of the PUM advise. Up front they had no idea what to 
expect as PUM is largely unknown in the country. Flalf of the companies interviewed consider a 
follow up mission and are prepared to pay the higher cost associated. They further mentioned that 
there is no alternative to PUM in terms of local expertise offered by consultants The reason to apply 
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for PUM is to get access to international best practice knowledge and management or production 
skills. 

7. Public support leverages investment by other entities that would otherwise not be forthcoming. 

8. Conditions attached to the cost-sharing project, or agency activities complementing it, are 
expected to have a positive influence on wider business operations, the business environment or 
other institutional factors. 

During the field visits we did not find strong indications that as a result of PUM other institutions 
stepped in to support the beneficiaries with additional capita! or otherwise. There were also no 
indications that other companies were stimulated to follow the recommendations of the expert. 
Some of the interviewed companies referred to some demonstration effects, but in general it was 
limited in the four countries visited. The support to BSOs and the sector programmes support by 
character more than one company, and their influence go beyond one company. 

In conclusion, virtually without exception it can be concluded from the interviews with the 
beneficiaries that in their perception the required expertise was not available locally, and if available 
(also for example internationally) at costs that the recipiënt SMEs could not afford. It should also be 
taken into account that foreign, in this case Dutch, advisors have a much better reputation than their 
local colleagues and are therefore preferred by the beneficiaries. This was confirmed by almost all 
interviewed entrepreneurs, who mentioned that they were in particular interested in the knowledge 
of experienced and independent Dutch colleagues and not in that of local consultants. Another issue 
mentioned was that local consultants are less interested in this type of short advisory assignments 
PUM is offering. 
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3.7 Is there a link between the number of countries and their status (LDC, LMIC, HMIC) 
and the relevance of the PUM programme? E.g. does a stronger focus on LDCs lead 
to a more or to a less relevant programme? 

As discussed in 3.4 there is a link between the OECD-DAC status and the relevance of a mission. 
Projects in LCDs and other low-income countries are less successful in reaching their objectives - as 
reported by beneficiaries to PUM directly - but more effective in closing gaps in knowledge - as 
measured by PRIME. This leads us to the conclusion that a country's level of development is 
negatively correlated with both the expected risk and the expected impact. Missions to more 
developed countries are less likely to fail in reaching the objectives of the mission, but their 
contribution to reaching the objectives of the Ministry is lower. In view of the definition of relevance 
for this evaluation (see ToR), it must be concluded that a stronger focus on LDCs would indeed lead 
to a more relevant programme. Considering PUM as an aid instrument with the aim to reduce 
poverty, it makes sense to focus on countries and also regions within these countries with lower 
levels of development. In these countries and regions, initial levels of knowledge and practices are 
generally lower, enabling PUM experts to make a greater contribution to closing gaps in knowledge 
and practices. The disadvantage of LDCs is that they present greater risks in the execution of 
projects. Initial levels of knowledge and practices could be too low or the local business environment 
is too constraining. 
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4 


Effectiveness 


4.1 How does PUM monitor the results of the programme? 

PUM collects data on the effectiveness of individual projects through debriefing and evaluation 
forms. Country coördinators and sector coördinators meet with the expert after the mission for 
debriefing. Both the expert and the beneficiary are asked to give a reaction on paper about their 
experiences. The beneficiaries are further asked whether the visit of the PUM expert was useful and 
whether he/she expects that his/her recommendations will have an impact on the company's 
performance and employment. With these monitoring activities, PUM checks the direct results of the 
visits. In other words, whether the missions have achieved their objectives and indeed resulted in 
practical recommendations. This internal monitoring system does not give an answer to whether the 
expected long-term results actually have been achieved. 

Whether the PUM support has actually resulted in increasing trade and/or investment relations 
between the supported SMEs and Dutch businesses is also not monitored accurately. Many experts 
continue to have relations and communication with the recipiënt SMEs, and there exist some verbal 
suggestions that this has resulted in business contacts, but these activities are not monitored 
systematically by PUM (PRIME, however, does monitor impact on trade and investment as we will 
discuss below). 

With the introduction of PRIME, the M&E of PUM has changed. In the previous system the expert 
and the beneficiary were both asked to estimate the impact of advice on the company's performance 
(e.g. turnover, profit, access to finance, etc.). In the current system, only the beneficiary evaluates 
this impact. Particularly the answers of beneficiaries are informative and may be seen as first signals 
of positive effects or the lack of effects, although one need to be aware of the possibility that 
beneficiaries do not give an honest and reliable answer to the questions. 

During the field visits we interviewed a selection of beneficiaries. We asked them about the effects of 
PUM on their company, in terms of knowledge, skills, turnover, profit, corporate social responsibility 
(CSR), gender equality, jobs and trade. They were requested to motivate their answer, enabling us to 
gain an understanding of how PUM affected the company. It is interesting to compare their answers 
with their reactions given in PUM's own questionnaire directly after the mission. In general, 
beneficiaries were more positive in this questionnaire - rating the (future) impact of PUM's advice 
good or even excellent - than in the interviews during our field visits. For example, there are two 
projects in our sample rated "very good" in the questionnaire, but with no impact on any of the 
indicators listed above. Another project was reported poorly-that means: the beneficiary rated the 
impact "sufficiënt" - but actually had a significant impact on the company. This leads us to the 
conclusion that evaluations by the beneficiary cannot be regarded as credible evidence of 
effectiveness, which in fact supports the decision to develop a monitor and evaluation system like 
PRIME. 

PRIME attempts to measure PUM's results, intermediate and long-term effects and impacts. It is 
however still in its early stages of development and therefore not capable yet to provide a full and 
representative measure of effects and impact. As soon as it possesses information about more 
cohorts of supported companies it will be in a better position to estimate the medium and long-term 
results of PUM. PRIME is collecting information about trade and investment relations, but at the 
moment of writing this report only information is available on the first cohorts. 
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4.2 Does PUM possesses credible evidence of the effectiveness of individual projects and 
if so, what are the conclusions? 

Comparing treated with non-treated firms 

Some first data of PRIME can already be used to estimate the impact of PUM on sales, profits, 
employment and export. For each of these outcome indicators it is possible to compare treated firms 
with non-treated firms. Counterfactual outcomes (changes in outcomes at non-treated firms) have 
been estimated by analysing changes in performance prior to the mission (e.g. comparing the 
situation two years before the mission with the situation one year before the mission). For this group 
of firms, changes in performance cannot be attributed to PUM, simply because there was no PUM 
intervention. They are considered being representative for the situation without a PUM 
intervention^^. Counterfactual outcomes can be compared with the outcomes of treated firms, 
analysing changes in outcomes before and after the mission (more precisely: the mission year and 
one year later^^). By comparing changes in outcomes between treated and non-treated firms, we 
gain some insight in the degree to which changes can be attributed to PUM. It should be stressed 
that this is not yet an 'other things equal' comparison, in the sense that for example changes in 
economie growth between the actual and counterfactual periods have not been taken into 
account^^. 

Comparing treated companies with non-treated companies, initial results suggest that participation 
in PUM has no substantial impact on increases in sales, profits and employment. The majority of 
assisted SM Es (75%) reported increases in sales, but for non-assisted companies this percentage is 
even slightly higher (81%). Analysing the change in profits, the difference between supported and 
non-supported companies is even smaller: 64 versus 66 per cent. This also applies to changes in 
employment: 62 versus 64 per cent. Only the share of companies that reported an increase in 
exports is substantially higher for treated firms: 14 versus 6 per cent. Due to small sample sizes^"* it is 
not yet possible to draw final conclusions on the contribution of PUM missions to changes in 
outcome. Flowever it is possible that a subset of participants in PUM do show substantial effects on 
economie performance, for example because they learned more from the PUM mission than other 
beneficiaries. This is analysed further below. 


We used data for each specific mission, not discriminating between first and follow-up missions. Yet, some firms may 
have had a PUM mission during the "counterfactual period". At the cost of a reduction in the sample size we did a similar 
exercise on first visits only for changes in profits. The conclusions from this exercise were about the same as reported in the 
main text. 

This is an approximation, not exactly just before and 12 months after. In the case of the employment, change occurring at 
least 6 months and at most 21 months after the mission, depending on in which month of the year the mission took place. 

Due to time restrictions and limited availability of data it was decided not to develop a model that does control for 
differences in macro-economic conditions. 

The impact on sales and employment change has been estimated by comparing changes in 176 treated firms with 225 
non-treated firms. For profit change, we compared a sample of 199 treated firms with 81 non-treated firms. For export 
change: 172 versus 72. 
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Figure 4.2: Profit change 



Data fmm PRIME surveys 


Figure 4.3: Change in exports 

100 T 

90 : 

80 

70 ^ 

60 i 

50 
40 
30 
20 

10 4 

0 4 - 

decrease 

Data f ram PRIME surveys 




Data f ram PRIME surveys 


41 















































Impact of attributable changes in practices on outcomes 


The effectiveness of PUM can also be analysed by developing statistical models^^ that test the 
influence of changes in practices attributable to PUM^® - as reported by beneficiaries - on outcome 
variables (e.g. without using a counterfactual, but using the attribution to PUM as stated by SMEs). 
This analysis results in some more convincing conclusions. One is that changes in practices 
attributable to PUM explain to some extent increases in sales. Changes in practices at PUM- 
supported firms (attributable to PUM) significantly (at 10% level) influence sales. If a company moves 
from the 25**^ to the 75'*^ percentile regarding changes in practices, changes in sales go up by 29%. 
However, the statistical model also reveals that changes in practices explain only a fraction of 
changes in sales considering the rather low adjusted (see Table 4.1). 

Similarly the change in practices has a significant effect on the change in profits and employment. For 
the change in profits the estimated coëfficiënt implies that as a firm moves from the 25**^ to the 75**^ 
percentile of the extent of change in practices, its profit change increases with about 15%^^. While 
the sample size is still relatively small, the linear regression results suggest a strong correlation 
between change in business practices attributable to PUM, and increase in employment after the 
mission. This relation is robust to the inclusion of a set of control variables. One point higher change 
in practices is expected to lead to a 3% higher (more positive) change in employment after the 
mission, and a change from the 25**^ to the 75'*^ percentile of practice change leads to an employment 
change of more than 12%. Among the Controls only firm age is strongly correlated with employment 
growth. As firms age they become more and more likely to show decreases rather than increases in 
employment after their PUM mission. Moving from the 25*^ to the 75**^ percentile of firm age 
decreases employment change by 6%. These findings support the idea that younger firms learn from 
PUM experts how to grow and expand, while older firms instead learn to become more lean and 
efficiënt. 


Table 4.1 ; Regressions for different sets of variables (bold indicate statistically significant) 


Change in Sales 


Change in 

Constant 

practices 

adj. 

N 

Coëfficiënt 

P-Value 

7.2615 19.1291 

(0.099) (0.190) 

0.0154 

114 

Change in Profits 


Change in 

Constant 

practices 

adj. R2 

N 

Coëfficiënt 

P-Value 

0.1508 0.8126 

(0.008) (0.000) 

0.0468 

127 

Change in employment 


Change in 

Firm age Constant 

practices 

adj. R2 

N 

Coëfficiënt 

P-Value 

3.1049 -0.4619 17.4462 

(0.002) (0.009) (0.000) 

0.0559 

226 


Based on a linear probability model (using OLS). 

Using the weight PUM knowledge impact indicator, introduced in the previous chapter. 

In the PRIME survey profit change is measured using a categorical indicator, which distinguishes changes in brackets of 
25%. For example it can distinguish a profit increase between 0% and 25% from a change between 25% and 50%. This 
results in a categorical variable with a total of 11 categories. In order to simplify the modelling process, it has been treated 
as a ratio scale variable, which can be modelled using linear regression. However it should be kept in mind that the 
coëfficiënt cannot be directly interpreted as a percentage change for a given change in the independent variable, but rather 
shows changes in numbers in the 25% brackets. 
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For a part of the PUM portfolio an increase in export share before and after the PUM mission may 
result from the changes in knowledge and practices induced by the mission^®. The only significant 
correlation found was between firm age and change in export share, which is shown in figure 4.5. 


Figure 4.5: Change in export share before and after the PUM mission, by firm age, 2012-June 2015 



Figure 4.3 shows that a change in export share is indeed related to firm age, but in a somewhat 
complex way. For young firms (between 2 and 4 years old) both increases and decreases of export 
shares are overrepresented. These changes decrease with age, up to the point that firms older than 
18 years hardly change their export share. As firms age, increases in export share become somewhat 
more likely than decreases, perhaps because of a selection effect (only relatively successful firms 
survive to live to an older age, and business success is likely to be correlated to increases in export 
share). In sum, the data suggests that at portfolio level no relation exists between PUM influence and 
changes in export share. This is, however, in line with the programme's theory of change: its main 
aim is to reduce poverty (by enhancing the performance of SMEs); the promotion of trade (whether 
or not with the Netherlands) is a secondary aim. 

Resuits from the fieid visits 

Beneficiaries in the four case study countries are less positive about PUM's impact on sales and 
profitability than the programme's impact on closing gaps in knowledge and skills. 54% of the 
respondents state that PUM has helped to improve sales, while 40% claims that PUM has helped to 
raise profits. In the interviews they admit that other factors - such as changes in economie conditions 
- have influenced changes in sales and profit levels to a great extent. On the other hand, there are 
many examples of successful companies stating that PUM made a significant contribution to 
improvements in sales and profits, rating this contribution as "very positive". In Ghana it was 
reported that in about 60% of the missions that PUM had a positive (30%) or even very positive 
(30%) effect on turnover, for example because the advice helped the company to improve the quality 
of its Products or to gain access to new markets. Flalf of the beneficiaries report a positive impact on 
profits, and as we would expect this impact is strongly correlated with the impact on turnover. To 
some extent this also applies to the impact on job creation (reported by 50%), although in some 
cases the advice of an expert resulted in a self-chosen reduction of employment due to efficiency 


38 Very little data was available for this relationship to be tested statistically. Export share is measured using a categorical 
variable with five categories (0%, 0-25%, 25-50%, 50-75%, 75%+). Given the limited sample size only a binary dependent 
variable could be constructed (has the export share increased or not?), and simple regressions using this binary dependent 
variable did not reveal any meaningful patterns. 
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measures and capita! investments. The Advent Press in Accra, fully owned by a religieus organization, 
is very positive about the impact of the PUM mission on sales and profit. lts staff reported: "Our 
output has increased because the productivity of machines has gone up. The quality of our jobs has 
improved and that is why sales went up, although the mission did not focus on sales. Therefore we 
attribute most achievements to PUM." 


Figure 4.6: Change in turnover and Profitability, % oftotal number ofcompanies interviewed 
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Source: interviews with 79 SMEs in Coiombia, Ghana Indonesia and Tanzania 

During the field visits there were other clear examples of successes. In Tanzania the PUM expert 
supported the Sluis Brothers to start a butchery in addition to their other meat producing activities. 
PUM experts assisted in various ways: drawing up the business model, gave advice on types of 
livestock, equipment, and on the facilities for smoked meat and cooling. Most of the (2"“^ hand) 
equipment was imported from the Netherlands. On the advice of PUM they are sourcing most of the 
meat processing ingredients from the Netherlands. The Sluis Brothers acknowledge that the technical 
inputs from PUM (three missions) have enabled them to set up a profitable business. They have 
introduced better livestock into the area and into the country. The company currently employs 20 
permanent staff and a number of casual workers. 

Figure 4.7: Empioyment generated as resuit of PUM advice 
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■ negative ■ No effect ■ positive ■ very positive ■ Don't know 
Source: interviews with 79 SMEs in Coiombia, Ghana, Indonesia and Tanzania 

In Colombia PUM advised Maquiempanadas S.A.S., a small company producing machines to make 
empanadas. It was established in 2010 grew rapidly to 15 employees in 2015. The PUM mission in 
May 2015 addressed issues related to HRM and product quality. The PUM expert reviewed the 
organisation, gave advice on HRM, management of the quality level of the manufacturing process 
and on customer relations management. With a view to increase e-commerce (especially to the 
USA), the layout of the website was reviewed as well, now also including English content. The PUM 
expert provided some follow-up recommendations via Skype. The recommendations were 
successfully implemented and as a resuit Maquiempanadas has continued to grow, with a further 
increase in empioyment (to 20 persons in 2016) and a 50% increase in profitability. 

The majority of SMEs reported empioyment increases, although often rather small. Figure 4.7 shows 
the impact on empioyment as reported by the interviewed companies during the field visits. It shows 
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that close to 50% of them experienced an increase in employment after the visit of the expert. They 
also mentioned that this increase can be attributed to the changes as recommended by the PUM 
expert. In about 40% of the cases employment did not increase, whereas in a few the number of 
employees was reduced as a result of the recommendations of the expert. According to the 
interviewed company staff there was no spill-over in terms of employment to similar companies in 
the sector, or they did not know. 

4.3 Is it, given available Information, possible to give an assessment of the effectiveness 
of the current geographical and sectoral spread? Would a different spread be more 
or less effective? 

In the previous section we discussed the impact of changes in practices on turnover, profits and 
employment. For the present evaluation question it was checked whether country status (OECD- 
DAC) has any influence on the conversion of practices into these business outcome variables. It 
turned out that country status has no significant impact on the outcome. This conclusion is, however, 
based on a relatively small sample from PRIME data. With the current data, we cannot state that a 
different geographic spread would be more effective. Based on the findings reported in the previous 
chapter, one would expect that missions to less developed countries and regions are more effective, 
assuming that the higher impacts on reducing gaps in knowledge and practices are translated into 
higher impacts on sales, profits and employment. The statistical model, however, shows that the 
change in practices is the only significant variable for explaining differences in outcome^®. The 
programme's effectiveness could only be enhanced by focussing on SM Es with a low initial level of 
practices and a strong potential for growth, no matter in what country. 

In view of the small sample size for the three models that explain effectiveness of the PUM 
programme (ranging from 114 to 226), it is rather difficult to assess the sectoral spread. Therefore, it 
has only been tested whether missions to production-oriented firms are more effective than to 
service and trade-oriented firms. This variable turned out to be insignificant in all three models. The 
only company characteristic that does matter is firm age as younger companies are expected to 
report higher changes in employment than mature firms. 

4.4 How does the effectiveness of sector specific programmes relate to the effectiveness 
of the individual interventions? 

With its support to BSOs that aim at creating an enabling environment for SMEs and with the 
organisation of sector specific programmes, PUM tries to reach more SMEs with a view to efficiently 
address common problems. BSO support^® is expected to strengthen the SME community when it 
concerns BSOs that represent and lobby for SME interests with policymakers and other stakeholders. 
Sector specific programmes include training or seminars for a group of SMEs with a view to inform 
them about how to tackle general problems they are facing. These programmes often also include 
short visits (less than the two weeks in the individual SME support) to the participants to assist them 
with their specific problems. 

Given the character of the BSOs and these sector programmes, the evaluation team faced serious 
difficulties to assess the effectiveness of this type of PUM activities. Support to a BSO as an 
organisation certainly improves its operations, as was mentioned several times during the visits to 
the case study countries. And it is most likely that SMEs benefit if the PUM support improves the 


Also interaction variables turned out to be insignificant. 

It should be noted that there is a difference between support to a BSO as organisation and support to a pool of SMEs in 
which the BSO is a partner. PUM's database does not distinguish between these two approaches. Only the latter is 
considered a sector specific programme. 
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(organisational) performance of the BSOs. The effectiveness of BSOs regarding their role in 
representing and promoting the SME sector and of the sector specific programmes depends, 
however, heavily on the extent to which the participating SM Es benefit from them, for which there is 
hardly any information. Evaluation of PUM's support to training institutions, such as the special 
programme of training institutions (e.g. Vehicle), face similar problems. The staff of the SMEs trained 
are expected to translate what they have learned into practice to the benefit of the SMEs. Similarly, 
performances of BSOs that defend the interests of their constituency should be judged on the basis 
of their successes. Although during the de-briefing sessions of the missions to these BSOs and 
programmes the experts make a judgement about the effectiveness of the support, there is hardly 
any information about these "second-line" effects. Neither PUM's own database nor the information 
collected by PRIME give any clue about the improved performance of the clients of the BSOs or of 
the special programmes after a visit of a PUM expert. Therefore we have to base our impressions on 
the verbal information from the interviews during the field visits with the representatives of the BSOs 
and SMEs, with other local stakeholders, and with PUM experts. 

The impression from these interviews is that the PUM missions have in general achieved the 
objectives formulated for these assignments. The interviewed BSOs reported that they have 
benefited much from the inputs of the PUM expert. The recommendations of the PUM experts were 
highiy appreciated and did not only eniarge the organisational knowledge but also improved the 
ways they operate to the benefit of their members (clients). As far as the interviewed representatives 
of the SMEs - some were member of such an organisation - could judge, they were positive about 
the performance of the BSO after the visit of the expert. The (admittedly small number of) 
interviewees are of the opinion that the contribution from PUM was beneficial for the clients of the 
BSOs. The picture of the special programmes is mixed. With the assistance of PUM, Danass Motors 
Company (based in Kumasi) organized a successful training in how to use on board diagnostics for 22 
members of an association of garage owners. They expressed that "Because of the PUM expert, we 
onderstand and do things better now. Our expectations were met: he explained and demonstrated 
auto computer principles. Before he came, knowledge of the auto computer was low." It is most 
likely that this course indeed contributed to a better performance of the participating SMEs and to 
more employment. The Ghana report also refers to a seminar on design which had hardly an impact 
on turnover or employment. Reason mentioned being that the seminar was mainly focused on 
transfer of knowledge and on the promotion of trade between Ghana and the Netherlands. For 
Indonesia it appeared to be virtually impossible to 'guestimate' the effects of the sector wide 
approaches, apart from, as said, verbal confirmation that they were useful. 

BSOs and the sector specific programmes play a, at the moment still limited, role in reaching 
companies with less than 10 employees. This category of companies is being served through training 
about common topics, sometimes followed by individual support. Given the poor conditions of these 
companies this approach fits well with the objective of poverty reduction. Unfortunately data is 
missing to determine its effectiveness for reasons explained above. 

4.5 Is it possible to enhance the effectiveness of PUM through a stronger collaboration 
with other PSD-programmes? 

As has been discussed in section 3.4 of this report the implementation of the recommendations of 
the PUM experts to the individual SMEs is often frustrated by external conditions, which reduces the 
effectiveness of the PUM programme. Most other PSD programmes are focused on reducing such 
external constraints. More collaboration in such a way that PUM advices on the internal problems of 
SMEs and that other PSD donor programmes focus on solving external problems could raise the 
effectiveness of the programmes of both partners. There are additional advantages as well. PUM has 
a rather unique position in the PSD donor community with its large pool of very experienced experts. 
Given this access to experts PUM could certainly play a role in other PSD programmes financed by 
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the Dutch government. Such a role could take different forms, varying from actual involvement of 
PUM experts in the execution of the other PSD programmes to providing them with information 
about SME development and potential winners (SMEs) in the partner countries these programmes 
are targeting. In return, PUM can learn as well from experiences of other PSD programmes that 
support SME development in developing countries and synchronize its activities with them in order 
to raise their effectiveness. In addition, PUM can benefit from information about for example the 
most constraining external factors that frustrate SME performance, which can be taken into account 
in the assessment of the applications for support. 

In spite of these examples collaboration is still exceptional. PUM's efforts to set up working relations 
with other PSD programmes has only be successful in a limited number of cases. PUM has taken the 
initiative for the PSD platform and pleaded for involving PUM experts in projects/programmes 
focused on the SMEs. This has resulted in collaboration with CBI in Peru, Colombia and Suriname, 
with SNV in Kenya, Rwanda and Uganda, and with SPARK in Somalia and Liberia. Reports about 
experiences with PUM experts are positive. 

In practice collaboration seems to be difficult for different reasons. First, depending of the 
alternative sourcing of expertise by the partners it could easily lead to crowding out of commercial 
service providers and therefore collaboration would conflict with PUM's principle of being additional. 
Secondly, it would require a more active exchange of information about what PUM is actually 
offering, doing and where. During the field visits, when discussing this issue it became clear that 
potentially interested PSD partners were not informed well about PUM's activities in the countries 
and therefore did not know how to involve PUM in their activities. Another obstacle for cooperation 
is that the existing PSD programmes are largely focusing on the provision of credits to SMEs through 
financial intermediaries. These financial institutions rely almost exclusively on their own supporting 
mechanisms to their borrowers and if not they often prefer commercial agencies. As a result, at the 
moment collaboration is exceptional, whereas it could increase the effectiveness of both PUM and 
other PSD institutions. 

4.6 How does the demand system work? Does PUM have the right incentives for 
ensuring the effectiveness of this system? 

The local representatives of PUM play a crucial role in identification and selection of SMEs and BSOs 
and in the promotion of PUM support to the individual SMEs and BSOs. Apart from idealistic motives 
related to helping SMEs in need for support, their incentive is the fee they receive for a successful 
application. Successful is first of all defined in meeting the criteria applied in the assessment of the 
application. These are in particular focused on relevance defined in this document as closing the gaps 
in knowledge and practices in the individual SMEs rather than on to what extent the support is 
expected to contribute to growth and employment. Similarly the country and sector coördinators 
who in the end approve the application or not, test it on the basis of the criteria of relevance. At best 
the additional documents explain that if the gaps are addressed successfully it will result in growth of 
the company and of employment and as such reducing poverty. From interviews with the local 
representatives and the country and sector coördinators it became clear that they implicitly assume 
that as soon as the problems of the SMEs are solved this will lead to a better performance and to 
more employment. And indeed this assumption is confirmed by the regression analysis presented 
earlier in this chapter. 

During the evaluation period, over 200 local representatives have been active for PUM in the target 
countries. In most countries there are more than one representative active, often responsible for a 
specific region within the countries. The regional distribution of the visits described earlier shows 
that some regions, and in the case of Indonesia and Ghana the most backward provinces, are not 
served. Main reasons mentioned being that less developed countries (regions) do not possess a well- 
developed SME sector, that there is a lack of supporting infrastructure and that the demand for 
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support from these regions is limited and also that therefore the risk for failure is great. Yet, given 
the objective of poverty reduction it would raise the effectiveness of PUM's system when more focus 
is given to these, admittedly from an operational point of view difficult, regions/countries. 

The representatives select the SMEs and BSOs on the basis of their private networks and they are 
often appointed because of these networks. This could result in a bias in the selection of projects. 
SME owners being part of such networks are often characterised as dynamic entrepreneurs and 
therefore well equipped to go after opportunities as they recognise them. Such an entrepreneurial 
attitude and the related commitment may indeed increase the effectiveness of the support. Yet, 
entrepreneurs who are less active in netwerking may miss the opportunity to benefit from the 
programme, while they might need it most. Potential effects on growth, employment and poverty 
does not play an explicit role in this selection process. 

As explained earlier in this report, PUM requires a contribution from the beneficiaries for the costs of 
a mission. They should cater for lodging and local transportation of the visiting PUM expert. In case 
there are follow-up missions they are required to pay an additional fixed amount of €700. In the 
philosophy of PUM these contributions increase the commitment and the ownership of the expert's 
visit and his/her recommendations. They are therefore prepared to introducé the changes proposed 
by the PUM expert. This indeed increases the effectiveness of the programme. From the interviews 
during the country visits it became clear that there is a positive relation between the success of the 
1^* mission reflected in a better performance of the company and the willingness to pay for the 
services provided. The companies that consider the 1^* visit a success had with hindsight be willing to 
co-finance the mission as well. 

Probably because the request for a 2"'* visit reflects the success of the first intervention PUM recently 
introduced as a pilot in Indonesia that beforehand there should be an agreement on two missions 
with the beneficiary paying the fixed amount for the second mission up-front. Interviewed 
entrepreneurs admitted that based on their experience during the 1^* mission a second could have 
been useful but that he/she was not able to raise the required co-funding. This was particularly the 
case for the relatively small companies. This information is an indication that the new policy might be 
biased against the smaller SMEs. 

Considering the large number of missions per year from a quantitative point of view the demand 
System works well. The relatively small number of proposals that are rejected during the assessment 
process in PUM headquarter indicates that the local representatives know well how to deal with the 
criteria that reflect PUM's policy priorities. As shown earlier in this document the majority of the 
missions actually executed have contributed to solve the gaps in knowledge and practices and as 
such contributed to the achievement of the ultimate objectives. From this point of view the demand 
System generates a more or less continuous flow of projects that in total contribute to the 
achievement of the objectives. Flowever, there are some weak parts. First, there is a certain bias 
against regions, where SMEs are confronted with larger bottlenecks. Support to this category of 
companies could be more effective. Although it was difficult to check during the visits to the case 
study countries, the compensation to local representatives for each successful mission has the risk 
that they go after the easy assignments without judging them in terms of effectiveness. Continuous 
monitoring of the activities of the local representatives is required to prevent them from following 
their own agenda. 

4.7 To what extent do business link interventions result in economie relations? 

Several indicators are available to measure the effectiveness of PUM with regard to promoting trade 
and export with the Netherlands. Business links are defined by the ToR as "a follow-up activity to a 
project of PUM and consists of a visit of a local company to the Netherlands with the aim to sell or 
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buy goods or services or to enter into joint ventures". They are registered by PUM as Business Link 
Project (BLP) or as training. BLPs explicitly aim to promote trade, while training sessions in the 
Netherlands first and foremost focus on the transfer of knowledge and skills. PUM decided to stop 
organising training sessions in the Netherlands from 2015 onwards. It also happens that PUM 
missions result in non-registered business links: companies visit one or more firms in the Netherlands 
but not through PUM. Beneficiaries, local reps, experts and other stakeholders sometimes refer to 
these visits as "business links" but there is no information on these links. And finally, PUM missions 
may result in trade without a business link. Experts assist firms with importing machinery or inputs 
from the Netherlands or with exporting their goods to the Netherlands. In the remaining answer to 
this evaluation question we will mainly focus on the effectiveness of the official business links as 
registered by PUM. An effective or successful business link results in trade between an assisted 
company and one or more Dutch firms. 

The PRIME expert survey asks experts whether he or she has been able to realise a successful 
business links, e.g. a business link that resulted in trade. This can be a registered business link project 
(BLP) or an non-registered business link. Non-registered business links may involve a visit to the 
Netherlands but not necessarily. Only 77 responses could be connected to official business links, 
including 18 training project. Only 6 of the 59 official business links were classified as "successful", 
and hence effective in realising trade. This is less than 10%. In most cases experts fill in the survey a 
couple of years (on average 2.5 years) after the mission took place. 


Table 4.2: Effectiveness of "official" PUM Business Links 




PRIME survey: successful BL? 



No 

Yes 

Total 

Official business links as registered by PUM 

BLP 

53 

6 

59 

Training 

17 

1 

18 


Total 

70 

7 

77 


Another question is whether missions result in trade, whether or not through a (formal) business 
link. Directly after the mission, experts are asked whether they have been able to realise trade 
between the SME and companies in the Netherlands. It is interesting to compare these answers with 
the results of the PRIME survey (see table 4.3). The results are somewhat remarkable. It could well 
be that PUM registers no trade directly after the mission, while PRIME does some years later (as in 
48 of the 703 cases: 6.8%). It also makes sense that plans for trade have actually not been 
implemented (228 cases: 32.4%). But how to explain the 40 cases in which PUM reported trade while 
PRIME did not? The only logical explanation is that respondents sometimes not qualify trade as a 
"business link". The figures also show that in about 12.1% of all missions registered by both PUM and 
PRIME, the expert reported a successful business link to PRIME, while in 6.5% all missions the expert 
reported trade between the company and the Netherlands to PUM (expert debriefing). 


Table 4.3: Successful business links versus (planned) trade as reported in expert debriefings 




PRIME: Successful BL? 




No 

Yes 

Total 

PUM: Trade? 

No 

350 

48 

398 

Planned 

228 

31 

259 

Yes 

40 

6 

46 


Total 

618 

85 

703 


Of all missions registered by PUM 6.2% lead to trade, based on the expert debriefings directly after 
the mission. In 35.4% trade is being planned, but PRIME has informed us that plans do not always 
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materialise. Many experts claim that they have been able to establish contacts between the 
beneficiaries and the Dutch business community, although these have not (yet) resulted in trade 
and/or investments. These observations of the experts are consistent with the findings during the 
field visits of the evaluation team to the case study countries. A few of the interviewed beneficiaries 
confirmed that they have visited the Netherlands and that they have learned a lot during this trip. 
Others mentioned that they have contacts with Dutch firms, usually established through mediation 
of the PUM experts. However, most of them mentioned that they do not have any contacts or 
business relation with Dutch firms. The larger sample allows us to analyse the influence of various 
factors on the development of trade. 


Table 4.4: Regressions business links for different sets of variables (bold is statistically significant) 



Employees 

Firm age 

Low 

income 

Upper 

middle 

income 

First 

mission 

Production 

Initial 

practices 

level 

Constant 

adj. R2 

N 

Coëfficiënt 

-0.0001 

0.0025 

0.0552 

0.0433 

-0.0532 

0.0408 

0.0142 

-0.0068 

0.0204 

413 

P-Value 

(0.588) 

(0.019) 

(0.054) 

(0.188) 

(0.086) 

(0.118) 

(0.225) 

(0.889) 




The regression results (table 4.4) show that firm age is significantly positively related to the 
development of trade, suggesting that missions to older firms are more likely to result in trade. 
Comparing the firm on the 25**^ percentile of firm age to the 75*^ percentile, the latter is expected to 
have a 3% higher chance of having trade. Also the dummy variable identifying a first mission as 
opposed to a follow-up mission is a significant predictor of trade development, showing a negative 
sign (about 5% lower chance). Hence follow-up missions are more likely to result in trade than 
missions to a new beneficiary. Missions to "other low income" countries show a 5.5% higher chance 
of trade development compared to lower middle income countries"*^. This last findings should 
however be treated with caution since it is not robust to changes in model specification. In sum, the 
regression results suggest that follow-up missions are most likely to lead to business links, and 
especially when older firms are targeted One would maybe expect that missions to production- 
oriented companies are more likely to result in trade, but this cannot be confirmed by the statistical 
model (the coëfficiënt is positive but insignificant). 

The field visits confirm that the establishment of business links in the form of increasing trade with 
the Netherlands has been limited. Most experts kept contact with their clients for some period, also 
with a view to mediate between them and potential Dutch partners, but this has not resulted in 
tangible flows of goods and services. This particularly applies to Tanzania and Colombia. The effects 
on trade with Indonesia and Ghana are positive but modest. In both cases 30% of the respondents 
declared that trade has increased. In Ghana, for example, much is being expected from the advices of 
a PUM expert to the design sector. In Indonesia, a leather company managed to enter the Dutch 
market after a PUM expert advised the company on how to improve both the organisation and the 
design of its leather bags. 


'■1 The category "other low income countries" comprehends a relatively small number of missions compared to other 
country categories. Kenya and Kyrgyzstan are the most visited countries in this category. 
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5 Efficiency 

5.1 Given available Information, is it possible to give an assessment of the efficiency of 
the management of the programme? 

The efficiency of PUM can be measured by analysing the ratio between outputs (missions) and inputs 
(budget, ftes, number of volunteers, etc.). Since PUM is a unique organisation it is difficult to 
compare its efficiency with other PSD organisations. The organisation is to a large extent run by 
volunteers which results in low operating costs compared to its outputs. It is possible, however, to 
assess changes in efficiency over time. Table 5.1 presents some indicators of efficiency, based on the 
information presented in chapter 2. The overall conclusion is that PUM has become less efficiënt in 
using human resources (staff and volunteers). While the organization has become bigger, the 
number of missions, business links and experts in the database declined. One could argue that 
volunteers are "free of charge" (no costs) and hence should not be included in output/input ratios, 
but also the efficiency indicators relating outputs to staff show a downward trend (less efficiency). 
There is one exception: country coördinators cover more countries in 2015 compared to 2012. 
Another observation is that PUM has succeeded in reducing the out of pocket costs per mission, 
notably because the costs of international travel declined. In this respect the organisation has 
become more efficiënt. 


Table 5.1: Efficiency indicators 



2012 

2013 

2014 

2015 

Experts/staff 

70.4 

64.4 

59.8 

57.2 

Missions/staff 

39.1 

39.3 

36.5 

35.2 

Experts/(staff + volunteers) 

20.1 

18.3 

16.7 

15.0 

Countries/country coördinator 

1.21 

1.24 

1.16 

1.29 

Missions/(staff + volunteers) 

11.2 

11.2 

10.2 

9.2 

Missions/local representatives"*^ 

8.0 

8.8 

8.2 

6.2 

Business links/staff 

4.6 

4.4 

3.6 

2.3 

Out of pocket costs/mission 

€3.019 

€ 2.607 

€ 2.577 

€2.317 


Another indicator of efficiency is the overhead compared to direct costs (costs directly associated 
with a mission) or total costs. For PUM, this percentage would be very high since the direct costs are 
very low, due to the contributions by volunteers. In order to get a more realistic estimate of PUM's 
overhead - compared to commercial organisations - we have valued the missions of the PUM 
experts applying commercial rates for the days they spend in the field taking into account an average 
length of a mission of 15 days. 


All local representatives registered in the PUM database in a year. 
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Tab Ie 5.2: Estimated casts for miss lans 



2012 

2013 

2014 

2015 

Assumptions about days per mission andfees 

Number of missions (A) 

1.875 

1.964 

1.900 

1.710 

Average number of days per mission (B) 

15 

15 

15 

15 

Total number of days spend on missions (A*B) 

28.125 

29.460 

28.500 

25.650 

Estimate average commercial daily fee (C) 

€ 1.000 

€ 1.000 

€ 1.000 

€ 1.000 

Total labour cost estimated at commercial daily fees (A*B*C) 

€28.125.000 

€ 29.460.000 

€ 28.500.000 

€ 25.650.000 

Out-off-pocket expenses, source annual accounts: 

- Voluntary staff reimbursement 

€ 297.402 

€325.029 

€ 340.878 

€357.331 

- Direct project costs 

€4.659.418 

€4.433.662 

€4.191.873 

€3.571.037 

- Representatives 

€703.171 

€361.912 

€363.524 

€309.523 

Total out-off-pocket 

€5.659.991 

€5.120.603 

€4.896.275 

€4.237.891 

Total costs missions valued at commercial rates 

€33.784.991 

€ 34.580.603 

€33.396.275 

€29.887.891 

Total overhead related to missions 

€ 6.208.483 

€ 7.043.937 

€7.417.706 

€7.898.353 

Overhead as % of direct mission costs 

18% 

20% 

22% 

26% 

Overhead as % of total costs 

16% 

17% 

18% 

21% 


The calculations show that, given the assumptions made, the ratio of overhead costs to mission 
related costs increased from 18% to 26% during the 2012-2015 period, which is in line with the 
trends reported above based on efficiency indicators. The ratio for 2015 compares high to what is 
common in a more commercial environment, in particular because we did not take into account the 
labour costs of volunteers who support the head office, such as country and sector coördinators. 
Given PUM's exceptional character of working with volunteers it is, however, virtually impossible to 
judge the absolute value of this ratio. Yet, the ratio shows a substantial increase during the 
evaluation period. One explanation is that PUM increased its efforts to mobilize non-DGIS sponsors 
for its funding. Another possible reason is that the number of missions declined due to the 
introduction of PRIME and the new ICT system. Expectations are that the number of missions will 
increase again in 2016, which would imply improved efficiency. 

While PRIME mainly focuses on collecting data on relevance and effectiveness, PUM's own M&E 
System also collects indicators of efficiency. This includes information on inputs and outputs as 
discussed above, but also on the efficiency as perceived by customers (beneficiaries). For example, 
beneficiaries are asked to indicate how they experienced the application process in terms of 
communication and speed of handling. The interviews with the beneficiaries in the case study 
countries revealed that most of them were satisfied with the accurateness of the process and many 
of them mentioned that they were surprised about the quick processing of the application/approval 
and also about the limited timespan of mobilizing the expert. Out of the 79 interviewed SMEs, 65 
gave PUM a score of good to very good. However, a small number of the interviewed persons 
complained about the fact that the expertise of the expert did not match with the problems to be 
addressed. In our sample, only a few beneficiaries complain about bureaucracy. Several experts and 
local representatives complained about the increasing (digital) paperwork, which in some cases 
resulted in delays or even unnecessary cancellations during the application process. They also refer 
to the emerging "survey fatigue" among experts and volunteers, who have to answer numerous 
questions for the M&E system of PUM. Better integration between PUM and PRIME and a critical 
review of current surveys could make the organisation more efficient'^^. 

Based on interviews with PUM and its stakeholders, and an analysis of its M&E system, it can be 
concluded that the procedures of PUM are straightforward. With help of the local representative the 
potential beneficiaries prepare the application for the assignment, which is sent to PUM 
headquarters. The assessment of these applications is being done by the sector and country 


Please note that also the external evaluation - of which this report is the result - is part of this M&E system. 
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coördinators and there is a check on the financial implications. This process also involves the 
selection of the expert. A relatively small number of applications is rejected because they did not 
meet the PUM criteria. An even smaller number of requests could not be rewarded because the 
needed expertise was not available within the pool of PUM experts. In the application process the 
role of the local representative is essential. He/she identifies the potential beneficiaries and supports 
the preparation of the applications. The pay-for-performance approach is not only an incentive for 
the quantity of applications but also for their quality, meaning meeting the PUM criteria. 

In conclusion, PUM collects sufficiënt information to measure efficiency over time, but it is difficult to 
compare efficiency with other organisations. Both output-input indicators and estimates of 
overhead/cost ratios (at commercial rates) suggest that PUM's management has become less 
efficiënt in using human resources during the years of the evaluation (2012-2015). If we only look at 
out-of-pocket costs, however, the organisation has improved its efficiency. Most beneficiaries are 
positive about the PUM organisation: the application process is well organised and there are no clear 
signs of avoidable bureaucracy, although the increasing number of procedures related to the M&E 
System require attention. 

5.2 Is the current geographical and sectoral spread efficiënt? 

The efficiency of the geographic and sectoral spread can be measured by analysing efficiency 
indicators as discussed above. For assessing the efficiency of the geographic spread we suggest using 
the average number of missions per local representative per year on a country level. The number of 
local representatives gives an indication of the amount of efforts PUM puts in a country, even if the 
local representative is paid on a "no cure no pay" basis'^''. The figures show that this indicator varies 
widely. Countries like Afghanistan, Montenegro and Zimbabwe are relatively inefficiënt, with on 
average 0.3 to 2.7 missions per local representative per year"*^. At the other extreme, local 
representatives in countries such as Nepal, Morocco and Suriname generate more than 25 missions 
per year. PUM's target is a minimum of 10 missions per local representative. The majority of the 
countries (44 out of 74) do not succeed in realising that goal, which suggests that the programme's 
efficiency could be improved by focusing on countries with a higher output per local representative. 
There is no clear relation between the development stage of a country (OECD-DAC status) and the 
efficiency of the programme, measured by number of missions per local representative'^®. The 
programme appears to more efficiënt in Latin-American countries (with on average 18.5 mission per 
local representative) than in African (14.4) and European (14.5) countries. Asia comes second with on 
average 16.2 missions. 

Although the indicator discussed above provides some useful insight in how PUM operates in 
different countries, its validity as an indicator of cost efficiency is questionable. Most costs of PUM 
relate to overhead on programme level or can be directly linked to missions (e.g. costs of travel). The 
marginal costs of being present in one additional country are very low. Country coördinator are 
volunteers and local representatives are paid for each successful mission only. The costs of having 
one extra country are limited to the travel costs of a country coördinator or a staff member (often 
only one visit a year). Therefore we conclude that - although there are differences in the output- 
input ratios - a different geographic spread would not necessarily improve PUM's efficiency. If PUM 
decides to end its activities in one country, the organisation has to intensify activities in another 


Local representatives can be associated with countries. For country coördinators this is more difficult because some of 
them serve more than one country. 

For Afghanistan and Zimbabwe the low efficiency can be explained by the difficult social and political circumstances in 
these countries, whereas the portfolio for Montenegro was scaled down because of its income position. 

LMICs perform slightly better, but the differences are not significant and not stable during the evaluation period. 
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country in order to keep the total number of missions at the same level (in line with the agreement 
with the Ministry). Country coördinators are expected to generate 50 to 60 missions a year; 
therefore, they sometimes have to cover several countries. When PUM (re)enters a new country, this 
requires additional investments: the country coördinator has to visit the country one or two times in 
order to check the potential for PUM missions. Stability in the list of countries raises efficiency 
because it allows PUM to build upon the experience of local representatives with handling the 
applications and smoothening the visits of experts. Such a strategy also creates an environment in 
which the local beneficiaries exchange Information about how to apply for support, and where they 
share their experiences with other SMEs. 

The arguments above also apply to sectors. The number of missions per sector (and sector 
coördinator) varies to a great extent. But the costs of having an extra sector coördinator are minimal. 
Of course PUM could still deelde to redefine sectors in view of a low number of missions, e.g. by 
making someone responsible for two (related) sectors (as happens with countries). However, such a 
measure should not be interpreted as a change in the sectoral spread. 

In conclusion: from an efficiency point of view, one could argue that it is better not change the list of 
countries covered by PUM too often, because such changes hardly influence the programme's 
efficiency and because of PUM's investments in local networks. There is also no need to reduce the 
number of sectors. 

5.3 Are sector specific programmes more cost-effective than individual interventions? 

As discussed in section 4.4 it is difficult to analyse the effectiveness of sector-specific programmes 
(involving visits to BSOs but also multiple visits to SMEs in the same sector) versus individual 
interventions. PRIME does not collect data about the results of visits to BSOs, and sector-specific 
programmes involving multiple visits to SMEs cannot be clearly distinguished in the data. In Ghana, 
for example, we found that some sector-specific programmes are registered as a visit to one single 
company: in that particular case, the expert had to visit several firms in only two weeks. In section 

4.4 we concluded that the effectiveness of sector-specific programmes is not necessarily higher than 
for missions to individual firms. Comparing outputs and outcomes with the costs of organising such 
missions, one could argue that the costs per assisted SME are generally lower for sector-specific 
programmes: their outreach is larger. A counter argument, however, is that such programmes tend 
to be more general: this particularly applies to sector approaches that only involve training. They 
assist companies with acquiring particular knowledge and skills, thus tackling general 
problems/issues that companies in the sector are confronted with, but not necessarily with solving 
specific issues in the companies. This means that sector programmes might be less effective in 
improving SME performance. Also in our (focus group) discussions with experts, local 
representatives, sector coördinators and country coördinators, there was no consensus on this issue: 
some clearly prefer a sector-oriented approach claiming that such an approach would be more 
effective, while others prefer individual interventions. Possibly this has to do with the fact that 
sector-specific programmes require different expertise. One-to-one missions require skills to assist 
and advise a company, while BSOs and sector-specific programmes often (also) require training skills. 
This particularly explains why experts have different views on this issue. 
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6 Forward Looking Questions 

6.1 Is there scope to strengthen the links between 'aid and trade' in the PUM 
programme, while maintaining a development focus? 

By combining insights from answers to evaluation questions discussed above, it should be possible to 
present an answer to this "forward looking question" about the potential synergies between aid and 
trade in the PUM programme. First, we can confirm that during the evaluation period PUM has 
mainly focused on the provision of aid, and not so much on developing trade. This particularly 
becomes clear when analysing the procedures in place to ensure the programme's relevance. The 
potential for trade with the Netherlands is not an issue in the application process, nor in the 
selection of countries or regions covered by PUM (e.g. by local representatives). The programme 
does focus on some sectors with export and trade potential, but in reality the programme is highiy 
demand-driven. As discussed in section 4.7, a relatively small portion (around 6%) of the missions 
leads to trade. From interviews we learned that most experts are driven by the motive to provide aid, 
and not by the motive to create trade, uniess these two can be combined: for example, when they 
can ship a (relatively cheap) second-hand machine from The Netherlands to the country and 
beneficiary in question. In several cases, however, experts advise to buy inputs from the Netherlands 
that are clearly out of reach for the beneficiary, due to limited budgets and low access to Capital. In 
these case aid and trade cannot be combined. 

In section 4.7 we also concluded that the country's level of development (OECD-DAC status) has no 
significant influence on the development of trade with PUM-assisted firms. This would imply that 
changes in the country mix are not the answer to the question on how to stimulate the combinatiën 
of aid and trade. Based on data, we find no convincing argument either to change the sector mix: 
there are no significant differences between production- and service-oriented firms. Several 
interviews and focus groups with beneficiaries, local representatives, sector coördinators and 
country coördinators, however, do suggest that trade and export are more likely to be generated in 
particular sectors. In view of the small sample that was used for the data analysis (only considering 
the difference between production and services, not between specific sectors), it is fair to state that 
PUM could indeed strengthen the link between aid and trade by focussing on sectors with "trade 
potential". And obviously, this implies that there are also differences between countries (so, within 
the broad categories of countries). For example, some interviewees indicated that countries that 
were recently taken from the list because the Ministry of Economie Affairs stopped funding PUM had 
a relatively high potential for combining aid with trade. Relevant factors are a country's economie 
structure, existing trade relations with The Netherlands and distance between the country and The 
Netherlands. 

6.2 Is it possible to interest Dutch business more in contributing to PUM? 

As discussed in chapter 2, PUM has been successful in acquiring funds from other sources than the 
government. So far, however, the interest of Dutch businesses in contributing to PUM financially has 
been marginal. Based on interviews with relevant stakeholders, we draw the conclusion that also for 
the coming years it is not likely that the Dutch business community will substantially increase its 
financial contributions to PUM. What some interviewees do claim, however, is that companies tend 
to become more interested in contributing to PUM by allowing their employees to do volunteering 
work for PUM. Through such in-kind contributions they invest in their human capita! but indirectly 
also in their relations with relevant sectors in developing countries. Such an approach would be 
particularly interesting for companies if PUM would indeed manage to enhance synergies between 
aid and trade (see 6.1), of course without compromising its development focus. It could raise the 
interest in establishing business links with the local companies that are supported. An additional 
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advantage is that private-sector involvement in PUM would rejuvenate its pool of experts, thus 
helping the organisation to increase knowledge and expertise on the most recent innovations and 
insights available for SMEs in developing countries. 

PUM could also be interesting for the Dutch business community in view of the weaith of information 
about SMEs in developing countries generated by PUM and PRIME (see 6.4). This information may be 
used to stimulate trade between the Netherlands and developing countries, whether or not through 
official PUM business links. Currently, the information about SMEs is hardly accessibility for external 
parties, but it might be feasible to change this situation, for example by communicating the 
information through PUM's mother institution VNO-NCW. 

PUM should be looking for opportunities to improve cooperation with Dutch businesses, but not to 
an uniimited extent. Several interviewees - both in the Netherlands and abroad - said that PUM 
should stick to its original mission. Other sponsors - in cash or in kind - are welcome but PUM should 
keep itself far away from more commercial activities such as participating in tender procedures that 
are also open for commercial advisory firms (notably in view of the programme's additionality). 

6.3 Would a closer collaboration of PUM with other PSD programmes lead to a more 
cost-effective approach? 

In section 4.5 we discussed the question whether cooperation between PUM and other PSD 
programmes would enhance effectiveness. Several arguments were presented in favour of such 
partnerships, notably the fact that other programmes play a role in improving the business 
environment in which the assisted companies operate. Changes in business practices attributable to 
PUM only predict a small part of the changes in performance as demonstrated by our statistical 
analysis. On the other hand, PUM's bottom-up approach could complement the top-down 
approaches of other programmes, thus enhancing their effectiveness. Also we mentioned the 
possibility that human resources (experts, volunteers) and data sources (information) could be used 
to their full potential, in PUM and other programmes. Based on interviews and focus groups, we find 
sufficiënt support for the claim that closer cooperation with other PSD programmes would lead to a 
more cost-effective approach. Whether this will actually materialise in practice, is more difficult to 
say, particularly because such cooperation is still rare and also because there is insufficiënt data 
about these exceptional cases'*^. Reports are positive, but additional research would be needed to 
verify the claims. Moreover, some obstacles need to be overcome. Some interviewees state that the 
differences between PUM and other PSD programmes are too large: they might be complementary 
but not compatible. Others emphasize that cooperation could harm the level-playing field, notably if 
it would lead to competition with local suppliers of knowledge. These obstacles need to be taken into 
consideration when discussing the possibilities for cooperation between PUM and PDS programmes. 

6.4 Is it possible to make better use of information, generated by PUM, for helping to 
strengthen the business climate in countries where PUM is active? 

PUM generates a weaith information on the characteristics and performance of SMEs in developing 
countries, as we discussed in various sections above (see, for example, 3.3 and 4.1). Notably through 
PRIME the programme provides insight in relevant indicators such as changes in knowledge and 
practices, turnover, employment, profitability, the share of export in turnover, etc. In the present 
evaluation we used data from the first cohorts to analyse factors that influence the performance of 
SMEs (see sections 3.4 and 4.2). When more data will become available, it will be possible to get an 


Collaboration with CBI in Peru, Colombia and Suriname, with SNV in Kenya, Rwanda and Uganda, and with SPARK in 
Somalia and Liberia. 
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even better understanding of the relation between company characteristics (e.g. size, age, 
ownership, sector, etc.), business environment characteristics (e.g. region, country, stage of 
development, etc.), interventions (e.g. PUM and business links, but possibly also other interventions) 
and business performance. PRIME also carries out qualitative case study research to gain better 
insight in contextual conditions (e.g. aspects of the local business environment) that influence the 
effectiveness of interventions. It is too early to draw conclusions on the usefulness of these 
qualitative case studies, but the outlook is promising. 

Information generated by PRIME can be used to enhance the performance of the PUM programme, 
e.g. by focussing on specific sectors and regions/countries or companies with particular 
characteristics. Based on these insights, additional criteria could be formulated in the application 
stage to make the programme more relevant and effective (e.g. by making a bigger contribution to 
strengthening the business climate). 

Furthermore, data generated by PUM and PRIME is potentially useful for other public and private 
actors: local governments, BSOs, educational and knowledge institutions, private investors and other 
PSD programmes. BSOs might use the data to learn how to support businesses effectively, while 
educational institutions could find out what gaps in knowledge and skills still exist. For local, regional 
and national governments - and other stakeholders - the data generate insights on how to 
strengthen the business environment. PUM could stimulate experts to use the data or set up specific 
programmes focusing on these target groups (such as Vehicle and the pilot programme that targeted 
governments). Or, alternatively, PUM could simply make the data available for other organisations. 
Several interviewees stated that PUM should share its data with other relevant parties, such as 
Netherlands embassies and other Dutch PSD programmes. This particularly seems feasible for 
anonymous data (e.g. on the level of sectors, regions and countries). Information about individual 
firms cannot be shared with other actors, uniess these actors become formally involved in PUM 
and/or beneficiaries formally allow PUM to share Information with one or more actors. 
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7 Conclusions and recommendations 

7.1 Conclusions 

Relevance 

PUM guarantees its relevance by applying adequate criteria for the selection of countries, sectors 
and beneficiaries. However, PUM is essentially "demand-driven" which explains why the programme 
cannot fully avoid giving support to missions that contribute little to the reduction of gaps in 
knowledge and skills. Relevance could be enhanced by making more explicit how PUM intends to 
ensure a contribution to closing gaps in knowledge and skills, e.g. by introducing additional/other 
criteria in the application stage. Currently the selection of countries, regions and beneficiaries is 
mainly driven by "chance of success" and not so much by "chance of impact". 

Mostly through PRIME, PUM has access to relevant data on the relevance of individual projects. The 
majority of the projects reach their objectives and contribute to closing gaps in knowledge and 
practices, as confirmed by the country studies. The programme has a clear focus on poverty 
reduction, but there are no mechanisms in place to focus on the poorest countries or regions within 
countries. We found no correlation between the presence of PUM (the number of missions per 1 
million inhabitants) and a region's or country's level of development (e.g. measured by average 
income). Missions to lower-income countries tend to involve more risks (the success rate is 
significantly lower), but if they are successful they generally result in significantly larger contributions 
to the programme's and therefore of the ministry's objectives (closing gaps in knowledge and 
practices with a view on the eradication of poverty). These results support a stronger focus on the 
least developed countries. 

Furthermore, results indicate that the initial level of knowledge and practices has a positive influence 
on the change in knowledge and practices, suggesting that PUM could become more relevant by 
focussing on SMEs with a high potential to absorb new knowledge. Another finding is that company 
size (the number of employees) has a positive influence on reported success (meeting the mission's 
objectives), but no influence on the contribution to closing gaps in knowledge and skills. In view of 
the large number of small and micro companies in developing countries, relevance could be 
enhanced by ensuring that these groups of companies are actually reached by PUM: directly or 
indirectly (e.g. through sector-specific programmes or missions to BSOs). The country studies reveal 
that missions to single companies rarely result in knowledge spill overs to "competing colleagues". 
Group- or sector-oriented programmes are therefore needed to make substantial contributions to 
poverty reduction. 

Additionality 

PUM checks to some extent whether subsidised missions to SMEs or BSOs are additional, following 
the criteria of DCED. Nevertheless we do observe some possibilities for improvement, particularly 
regarding the check on the SME's ability to finance similar advice by own means. In a few exceptional 
cases entrepreneurs own several companies and seem to be able to invest in their own company. 
Entrepreneurs often argue that PUM advice is unique: they highiy value the advice of a foreign 
expert. They claim that local or international consultants are not willing or able to provide the same 
kind of advice, or only at costs they cannot afford. Results from the field visits suggest that some 
companies are willing and able to cover some of the costs (e.g. the international ticket of the expert), 
but in general they cannot afford a commercial alternative. The programme's additionality could be 
improved by focussing on companies with low access to capita! (an indicator for the ability to pay) 
and low initial levels of practices (an indicator for the ability to improve practices without support). 
This combination is mainly found among younger firms as one would expect. 
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Effectiveness 

With the creation of the PRIME partnership, PUM has made a significant step forward in collecting 
data on the effectiveness of its activities. Since PRIME started collecting data on the performance of 
PUM-assisted firms only a couple of years ago, the current evaluation is based on a relatively small 
sample. Nevertheless PRIME data has already proven its usefulness to measure the programme's 
effectiveness. The most interesting finding is that changes in practices attributed to PUM - as stated 
by beneficiaries - explain to some extent changes in performance (turnover, profit and employment). 
This finding is supported by insights from the case studies which present several more qualitative 
examples of companies that have been able to enhance their performance with the assistance of 
PUM experts. 

Of course, we must realise that numerous other factors influence such changes in outcome variables, 
but by combining PRIME data with our own research we find sufficiënt evidence that PUM experts 
are able to make a change in only two weeks' time. Both sources also suggest that the impact on 
employment is somewhat lower than the impact on sales and profit. PUM missions to younger firms 
are more likely to result in new jobs than missions to mature firms: a possible explanation is that 
PUM experts advice young firms on how to grow, and mature firms on how to become more lean 
and efficiënt, which sometimes involves a reduction of jobs. We found no significant influence of 
country status on effectiveness and also there is no difference between missions to production- 
versus services-oriented companies. More data would be needed to explore differences in 
effectiveness between specific sectors (e.g. within production). 

Sector-specific programmes are not by definition more effective than individual interventions, but 
there are some strong arguments in favour of such programmes. Sector-specific programmes will 
only be effective if they manage to translate changes in knowledge into changes in practices. This can 
be done by offering hands-on training sessions but also by combining seminars or trainings with 
missions to individual companies. Furthermore we conclude that PUM hardly collects data on sector- 
specific programmes. BSOs are not included in PRIME and it is not clear from the data if a single 
mission is part of a sector or group approach. 

Collaboration with other PSD programmes could enhance PUM's effectiveness, but there is no data 
available to make a systematic comparison between collaborative and non-collaborative approaches. 
Arguments in favour of collaboration are strong though: PUM could benefit from cooperation with 
PSD programmes that aim to reduce constraints in the business environment. 

Although it is not the primary aim of PUM to promote trade, it does occasionally succeed in setting 
up trade relations between assisted SM Es and Dutch companies. However, Business Links (visits of 
SMEs to companies in the Netherlands, registered and managed by PUM) are not necessarily needed 
to realise this secondary aim. Quite the contrary: Business Links rarely result in actual trade relations. 
The effectiveness of Business Links as generators of trade is questionable. 

Efficiency 

Given PUM's unique characteristics, it is nearly impossible to compare its efficiency with other 
organisations. PUM does collect, however, data that can be used to measure changes in efficiency 
over time, thus making it possible to analyse potential impacts of changes in policy (e.g. the number 
of countries and sectors targeted) on efficiency. Overall the conclusion is that PUM has become less 
efficiënt in using human resources, possibly due to its intensified search for other sources of funding 
and the introduction of a new ICT system. Most efficiency indicators show a downward trend 
between 2012 and 2015. On the other hand, the out-of-pocket per mission declined, which suggest 
increasing cost efficiency. Although the number of missions per local representative can be used to 
analyse geographic differences in efficiency, the validity of the indicator is questionable. It is true that 
more than half of the countries do not meet PUM's target (at least 10 missions per local 
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representative), but it is not likely that changes in the geographic spread will make the organisation 
more efficiënt. The main reason is that the marginal costs of being present in an extra country are 
minimal. The same reasoning can be applied to the sectoral spread. Overall, PUM is considered an 
efficiënt organisation. Beneficiaries are satisfied with the communication and speed of handling in 
the application stage. There are no clear signs of avoidable bureaucracy, although the increasing 
number of procedures related to PUM's M&E system require attention. 

Without volunteers PUM would not exist. They run to a large extent the organisation at home and 
travel around the world to advise and support companies. Their primary motive is their wish to share 
knowledge and expertise with their colleagues in less well-off countries and as such contribute to the 
reduction of poverty. They do this with great passion as was clearly illustrated during interviews and 
focus group discussions. Their commitment is mentioned by virtually all beneficiaries interviewed in 
the case study countries. These beneficiaries were not only impressed about the professional 
attitude of the visiting PUM experts, but in particular also about their drive to improve the 
performance of company demonstrated during and after their visits. 


7.2 Recommendations 

• Continue the PRIME partnership. The data is useful for measuring relevance and 
effectiveness. 

• Reconsider criteria for application. Include additional criteria to check on additionality, 
relevance and effectiveness. Focus on impact. 

• Further streamline the M&E system by reducing the "survey burden" for volunteers and 
beneficiaries: avoid asking questions twice (e.g. by PUM and PRIME). 

• Develop mechanisms that ensure a focus on poverty, in the selection of countries and 
regions within countries. 

• Explore opportunities to develop more sector-specific programmes and monitor their 
(cost)effectiveness in comparison with individual missions. Also look for other modalities to 
reach small and micro companies. 

• Strengthen cooperation with other Business Support Programmes, e.g. by sharing data 
generated by PRIME and sharing human resources. 

• Stick to the original mission of PUM: two-week missions of experts to SMEs. Avoid 
involvement in commercial activities/partnerships. 

• Enhance cooperation with Dutch firms with the assistance of PUM's mother institution VNO- 
NCW. Update the pool of experts by including active professionals from Dutch companies. 

• Continuously monitor local representatives. They are the pillars of the PUM programme. 
Keep on training and instructing them, and terminate their contract with PUM if needed. 

• Invest in promoting and marketing PUM, not only in target countries but also in The 
Netherlands. 
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Annex 1: PUM's portfolio 


Tabla Al: Missions by continent, 2012-2015 (June) 



2012 

2013 

2014 

2015* 


abs. 

% 

abs. 

% 

abs. 

% 

abs. 

% 

Africa 

688 

36.7 

780 

39.7 

756 

39.8 

351 

40.3 

Asia 

573 

30.6 

575 

29.3 

523 

27.5 

232 

26.6 

Europe 

315 

16.8 

296 

15.1 

287 

15.1 

124 

14.2 

Latin America 

299 

15.9 

313 

15.9 

334 

17.6 

164 

18.8 

Total 

1875 

100.0 

1964 

100.0 

1900 

100.0 

871 

100.0 


Source: PUM database, *preliminary 


Tabla A2: Number of missions by country type, 2012-2015 (June) 



2012 

2013 

2014 

2015* 


abs. 

% 

abs. 

% 

abs. 

% 

abs. 

% 

Least Developed 

447 

23.8 

563 

28.7 

619 

32.6 

286 

32.8 

Other Low Income 

40 

2.1 

37 

1.9 

36 

1.9 

20 

2.3 

Lower Middle Income 

937 

50.0 

933 

47.5 

812 

42.7 

394 

45.2 

Upper Middle Income 

451 

24.1 

431 

21.9 

431 

22.7 

166 

19.1 

Other 

0 

0.0 

0 

0.0 

2 

0.1 

5 

0.6 

Total 

1875 

100.0 

1964 

100.0 

1900 

100.0 

871 

100.0 


Source: PUM database, *preliminary 


Figure Al: Number of Projects per mln inhabitants by country in Africa 
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Figure A2: # Projects per mln inhabitants by country in Asia 
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Figure A3: # Projects per mln inbabitants by country in Europe 
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Table A3: Missions by type, 2012 to 2015 (June) 



2012 

2013 

2014 

2015* 

Total 

Firms 

1,543 

1,595 

1,599 

702 

5,439 

..SME 

1,540 

1,593 

1,594 

699 

5,426 

..big company 

1 

1 

0 

2 

4 

..transformation company 

2 

1 

5 

1 

9 

BSO 

320 

349 

285 

146 

1100 

..sector organisation 

30 

42 

36 

18 

126 

..chamber of commerce 

16 

18 

16 

9 

59 

..employees organisation 

1 

5 

3 

0 

9 

..healthcare organisation 

7 

7 

4 

0 

18 

..infrastructural organisation 

30 

19 

16 

7 

72 

..institution ofvocational education 

163 

178 

120 

60 

521 

..research and/or development organisation 

52 

52 

36 

22 

162 

..other 

21 

28 

51 

25 

125 

..unknown 

0 

0 

3 

5 

8 

Government 

6 

8 

7 

5 

26 

Other 

6 

12 

9 

11 

38 

Unknown 

0 

0 

0 

7 

7 

Total 

1,875 

1,964 

1,900 

871 

6,610 


Source: RUM database, *preliminary 


Table A4: Missions to firms by cluster, 2012-2015 (June) 



2012 

2013 

2014 

2015* 

Agriculture & Horticulture 

154 

170 

153 

79 

Building & Construction Trade 

58 

65 

78 

28 

Business Support & Management 

162 

180 

161 

67 

Chemical & Synthetic Materials 

90 

93 

80 

37 

Civil Service & Government 

3 

1 

1 

0 

Electrotech. Industry& Engineering 

26 

23 

26 

9 

Energy & Water & Environment 

67 

54 

59 

28 

Financial Institutes 

23 

26 

24 

10 

Food & Beverages Production 

189 

221 

198 

102 

Flealthcare 

53 

61 

55 

18 

Metal Industry 

120 

85 

73 

36 

Paper & Cardboard & Packaging 

15 

4 

13 

3 

Publishing & Printing 

49 

38 

41 

13 

Stockbreeding & Fisheries 

163 

183 

194 

87 

Textile & Leather 

49 

46 

72 

28 

Tourism & Flotels & Catering 

162 

190 

233 

94 

Trade 

43 

34 

35 

23 

Training & Education 

41 

39 

27 

10 

Transport & Logistics 

22 

23 

23 

10 

Wood Trade & Processing 

54 

59 

53 

20 

Total 

1,543 

1,595 

1,599 

702 


Source: RUM database, *preliminary 
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Table A5: Missions to firms by goal, 2012-2015 (June) 



2012 

2013 

2014 

2015* 

Total 

Technical 

952 

1,022 

948 

455 

3377 

Education and Training 

339 

358 

397 

180 

1274 

Export 

63 

45 

58 

21 

187 

Financial / business plan 

180 

180 

213 

80 

653 

General management 

588 

585 

619 

290 

2082 

Marketing 

353 

343 

386 

168 

1250 

Seminar 

16 

16 

17 

11 

60 

Total 

1,543 

1,595 

1,599 

702 

5,439 


Source: RUM database, *preHminary 


Table A6: Misslons to firms byfirm size (number of employees) 2012- June 2015 



2012 

2013 

2014 

2015* 


abs. 

% 

abs. 

% 

abs. 

% 

abs. 

% 

<25 

686 

44.5 

701 

43.9 

734 

45.9 

326 

46.4 

25-100 

610 

39.5 

671 

42.1 

643 

40.2 

260 

37.0 

100-500 

230 

14.9 

212 

13.3 

203 

12.7 

107 

15.2 

500+ 

9 

0.6 

7 

0.4 

15 

0.9 

7 

1.0 

Unknown 

8 

0.5 

4 

0.3 

4 

0.3 

2 

0.3 

Total 

1,543 

100.0 

1,595 

100.0 

1,599 

100.0 

702 

100.0 


Source: RUM database, *preliminary 


Table A7: New versus follow-up missions, 2012-2015 (June) 



2012 

2013 

2014 

2015* 


abs 

% 

abs 

% 

abs 

% 

abs 

% 

New 

1,288 

68.7 

1,329 

67.7 

1,415 

74.5 

669 

76.8 

follow-up 

587 

31.3 

635 

32.3 

485 

25.5 

202 

23.2 

..Ist follow-up 

317 

16.9 

362 

18.4 

261 

13.7 

101 

11.6 

..2nd follow-up 

105 

5.6 

119 

6.1 

92 

4.8 

22 

2.5 

..3rd or later follow-up 

165 

8.8 

154 

7.8 

132 

6.9 

79 

9.1 

Total 

1,875 

100 

1,964 

100 

1,900 

100 

871 

100 


Source: RUM database, *preliminary 
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Annex 2: Evaluation Methodology 

Annex 2.1 Purpose of the Evaluation 

According to the ToR^® the purpose of the evaluation is to assess the: 

1. (continuous) relevance of the PUM programme, including the poverty focus as well as the 
relevante for the aid and trade agenda; 

2. improvement of the monitoring and evaluation function of PUM; 

3. adequacy of available management information for monitoring and evaluation purposes, in order 
to ensure the relevance, efficiency and effectiveness of the programme; 

4. effectiveness of the PUM-programme during the period 2012-2015; and 

5. efficiency of the management and programme itself. 

In line with this purpose the research questions formulated in the ToR focus on both the 
management and the processes/procedures of the programme, including the established monitoring 
and evaluation system in PUM (PRIME and PUM's in-house monitoring activities), as well as on the 
effectiveness of the activities that support the SM Es in PUM's target countries. In addition, the ToR 
formulate a number (4) of what it calls forward looking questions. 

The ToR present the Theory of Change, which explains the logic behind the PUM-programme'*®. This 
Theory of Change forms the basis of the results chain of the programme as reflected in Figure 8 
below. For the inputs, activities and outputs this figure makes a distinction between the programme 
as a whole and the interventions of the programme. It summarizes the logical sequence from inputs, 
activities and outputs of respectively the PUM organisation and of the missions of the PUM experts 
to support/advice SMEs and BSOs, etc. to immediate, intermediate and ultimate outcomes and long¬ 
term results of the programme. 

At the programme level (PUM head quarter) PUM has access to funding allowing it to manage and 
run its programme. In addition the main inputs comprise activities of its own regular staff and time 
made available by voluntary experts from the Netherlands. PUM has established a pool of senior 
experts covering a wide range of expertise. On the basis of the requests from SMEs in developing 
countries PUM selects the experts who visit and advise these SMEs on a large variety of issues 
related to (running) their businesses. As a result it is expected that the performance of these SMEs 
improves, their numbers of employees increase and as such contribute to employment and economie 
growth in the targeted countries. To realise similar targets PUM also aims at establishment of 
business links between SMEs in developing countries on the one hand and businesses in the 
Netherlands and other countries in Europe on the other hand. Similarly on request PUM (experts) 
supports local Business Support Organisations (BSOs) in the target countries. 


See "Evaluation Foundation Netherlands Senior Experts (PUM) 2012-2015", 28 August 2015 

See Annex attached to the Terms of Reference of this evaluation study: "Evaluation Foundation Netherlands Senior 
Experts (PUM) 2012-2015", 28 August 2015 
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Annex 2, Figure 1: Results chain of the RUM programme 



Inputs 1 Activities | Outputs | Outcomes | Long-term results 

Programme level 

• Policies 
regarding PUM 
programme 

• Funds made 
available by external 
sponsors, e.g. 
DDE/Ministry of Foreign 
Affairs, DSO/Ministry of 
Foreign Affairs; others 

• Management 
of the programme 

• Time regular 
staff of PUM 

• Moblllsatlon 
of voluntary Dutch 
experts 

• Recrultment 
of local representatives 

In target countries 

• Organisation 
of short term advisory 
misslons of experienced 
Dutch experts 
(entrepreneurs) to SMEs 

In target countries 

• Organisation 
/ establishment of 
business links 

• Organisation 
of support / advice to 
BSO's in target 
countries 

• Organisation 
of seminars (support to 
multiple SMEs) 

• Assessment 
of the applications, on 
the basis of strict 

criteria 

• Matching 
demand and supply of 
expertise 

• M&Eofthe 
programme activities 

• #ofvisitsof 
experts advising SMEs in 
target countries 

• # of business 

links between 
companies in the 
Netherlands/EU and In 
target countries 

• # of BSO 

support activities in 
target countries 

• # of Seminars 

(advice to BSOs and 
vocational education) 

• # of Dutch 

experts In the database 

• # of local 

representatives In 
database 

Immediate Outcome: 

• Improved 
knowledge of the 
supported SMEs 

• Improved 
knowledge of supported 
BSOs 

Intermediate Outcome: 

• Changes in 

SME practices 
(Capacities, behaviour, 
CSR) 

• Changes in 

BSO services (capacities, 
behaviour, CSR) 

Ultimate Outcome: 

• Improved 
performance supported 
SMEs 

• CSR practices 

• Improved 
productlon capacity In 
varlous sectors In target 
countries 

• Increased 
exports to the 
Netherlands/ EU 

• Economie 
growth in developing 
countries 

• Increased 
employment In 
supported SMEs, In 
particular for poor and 
vulnerable groups 

• Sustainable 
Economie Development 

• Favourable 

conditlons for 
international economie 

activities 

• Presence of 
Dutch companies and 
Products on foreign 
markets 

Interventlon level 

• Guidelines on 
selection of projects of 
PUM programme 

• Funding 
made available through 
PUM 

• Experts made 
available through PUM 

• Time of 

voluntary experts 

• Time of 

country representatives 

• Time regular 

staff of PUM 

• VIsIts of 
voluntary Dutch experts 
to SMEs In target 
countries 

• Remote 
coaching of SMEs by 

PUM experts 

• Establlshing 
of business links 
between companles In 
the Netherlands/EU and 

In target countries 

• # of BSO 
support activities in 
target countries 

• # of Seminars 
(advice to BSOs and 
vocational education) 

• advice to 

individual SMEs in 
target countries 

• Remote 
coaching of individual 
SMEs by PUM experts 

• business links 
between companles In 
the Netherlands/EU and 

In target countries 

• support/ 
advice to BSOs in target 
countries 

• Seminars 
(advice to BSOs and 
vocational education) 


Annex 2.2 Evaluation matrix 

The evaluation matrix presented in the inception report specifies the criteria and sources of 
Information related to the research questions as given in the ToR. At the programme level the 
evaluation will provide a reconstruction of the PUM programme and its organisation and processes. 
This reconstruction will focus on the 2012-2015 period (see among others: "Beleidsplan PUM 2012- 
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2015", July 2011), but if relevant will also take into account the programme prior to this period. The 
ToR specify in detail the research questions for the evaluation. The reconstruction will also take into 
account the role and reactions of the stakeholders of the PUM-programme. Among them are: 

• Ministry of Foreign Affairs 

• PUM management; 

• Royal Netherlands Embassies; and 

• VNO/NCW (Dutch employers' organization). 

Given its purpose referred to above the evaluation will make an independent assessment, taking into 
account PUM's processes/procedures and the extent to which these processes/procedures have 
contributed to the effectiveness/impact of the programme applying the evaluation criteria as 
specified in the ToR.“ 

Sources of information 

In the evaluation matrix, we refer to the following "sources of information": 

1. Documents PUM: internal and external policy documents, including annual reports, policy plans, 
documents on processes, procedures, criteria and M&E, information on the PUM website, 
contracts with DDE, etc. 

2. Data PUM/PRIME: all quantitative data available via the databases of PUM and PRIME. More 
specifically we refer to: 

a. PRIME Survey beneficiaries, in particular to specific questions in the survey; if specific 
questions are not included in the survey they can be added (indicated as added 
question). 

b. PRIME Survey experts, in particular to specific questions in the survey; if specific 
questions are not included in the survey they can be added (indicated as added 
question). 

c. PUM Portfolio, in particular specific information on beneficiaries, experts, missions and 
other activities. 

d. PUM Survey business links, among Dutch firms that participated in business links; to be 
carried out by PUM in early 2016. 

3. Documents PRIME: policy briefs, newsletters, information on the PRIME website, (draft) 
reports/papers. 

4. Case studies PRIME: interview reports, (preliminary) analysis of results by PRIME 

5. Interviews PUM: interviews with the staff of PUM, e.g. Alex Meerkerk, Thijs van Praag and staff 
responsible for HR allocation 

6. Interviews PRIME: interviews with the staff of PRIME, e.g. Giel Ton and Karen Maas. 

7. Other interviews NL: interviews with other stakeholders in the Netherlands, such as 

a. The Ministry of Foreign Affairs (DGIS-DDE and DGIS-DSO) 

b. The Ministry of Economie Affairs 

c. VNO-NCW 

d. Dutch NGO's 

e. FMO 

8. Interviews beneficiaries: semi-structured interviews with different types of beneficiaries from 
different sectors in selected countries: 

a. SMEs 


See section 3.2 (ToR) and the OECD/DAC Criteria for Evaluating Development Assistance. 
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b. BSOs 

c. Special programmes, such as VEHICLE 

9. Other interviews abroad: semi-structured interviews with other stakeholders in selected 
countries: 

a. Royal Netherlands Embassies 

b. Actors involved in sector-specific programmes, e.g VEHICLE 

10. Survey and interviews local representatives: semi-structured interviews with local 
representatives in the selected countries 

11. Focus groups (or interviews) with volunteers of PUM, more specifically with 

a. Sector Coördinators 

b. Country Coördinators 

c. PUM Experts 

12. Interviews country coördinators: semi-structured interviews with the country coördinators of 
the selected countries 

13. Interviews PUM Experts: semi-structured interviews with PUM experts with experience in the 
selected countries 

Different types of EQs 

The aim of the present evaluation is to assess the relevance, additionality, effectiveness and 
efficiency of the PUM programme between 2012 and 2015 and to review how PUM could improve 
these performance indicators. The evaluation questions (EQs; as formulated in the ToR) suggest 
various strategies for improvement that have to be explored. Whereas some EQs clearly relate to the 
programme as a whole, other EQs refer to "individual projects" or "individual interventions". These 
include the various "modalities" of PUM: the missions to SMEs and BSQs, business links and sector- 
specific programmes (which consist of various trainings and missions to both SMEs and BSQs). Some 
EQs refer explicitly to these specific types of projects or interventions. 

The EQs for the present evaluation are grouped into four categories: 

1. EQs about the relevance and additionality of the programme 

2. EQs about the effectiveness of the programme 

3. EQs about the efficiency of the programme 

4. So-called "forward looking questions" 

For the first three groups of EQs, the evaluation matrix presents definitions of the evaluation criteria 
(relevance, additionality, effectiveness and efficiency) that can be applied to PUM, and formulates 
specific questions (judgement criteria) that help us to assess the performance of PUM on these 
aspects. Furthermore it defines, for each EQ, a number of indicators that allow us to make an 
independent and evidence-based assessment. In the most right column of the matrix we specify the 
sources that provide the information needed for composing the required indicators, answering the 
questions under "judgement criteria" and answering the EQ. For the "forward looking questions" it is 
not possible to define judgement criteria and indicators, but the matrix does specify the sources of 
information, which includes indicators of other EQs. 
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Annex 2.3 Operationalization of evaluation questions 


Relevance and additionality 

Relevance is defined as "contributing to Ministry's policy objectives, solving gaps in knowledge and 
skills in SMEs in developing countries and thereby contributing to the creation of jobs, especially for 
poor and vulnerable groups" (ToR). 

This definition refers to the objectives of the Dutch Ministry of Foreign Affairs as specified in its policy 
letters^^. These objectives are the eradication of extreme poverty, sustainable and inclusive growth 
across the globe and success for Dutch firms. This implies that the relevance of PUM should also be 
considered in relation to the Dutch "aid and trade" agenda, as it combines the provision of aid to 
SMEs in developing countries with the promotion of trade between these SMEs and Dutch 
companies, notably through business links. The ToR definition of relevance does not refer to the 
promotion of trade explicitly, but we suggest including it in the definition also in view of several EQs 
in this direction (e.g. under forward looking questions). 

The phrase "solving gaps in knowledge and skills in SMEs in developing countries" pertains to the 
intended outputs and outcomes of the programme, while "the creation of jobs" represents one of 
the main longer term results. Clearly the concept of relevance should be seen in relation to the 
programme's theory of change (the logic behind the programme) and the "results chain" we 
developed to assess the performance of PUM. In our view the "creation of jobs" is a long-term result. 
On the basis PRIME data a first estimate of direct employment growth following PUM missions will 
be made. PRIME will be able to do more precise estimates of employment effects when Information 
about multiple cohorts become available. The quantitative results will be triangulated and 
complemented with qualitative Information during the field visits. There is more Information 
available about the intermediate effects of PUM, such as the change in knowledge and skills in the 
supported SMEs and whether these changes have indeed resulted in a better performance of the 
supported SMEs. The first rounds of questionnaires executed by PRIME provide Information about 
these direct results. 

The last part of the definition emphasizes the Ministry's focus on "poor and vulnerable groups". On 
the one hand it implies that PUM is expected to support SMEs and economie growth in the least 
developed and most fragile countries. On the other hand it also suggests that the programme should 
bring benefits to the lowest income groups within these countries. This second interpretation is in 
line with the Ministry's objective to promote inclusive growth. In addition to inclusiveness we suggest 
paying attention to the contribution of PUM to sustainable growth, which implies also care for the 
environment. Both inclusiveness and sustainability are addressed in PUM by stimulating SMEs to 
adopt principles of corporate social responsibility (CSR). In addition to the Information collected by 
PRIME the case studies should shed light on the role of PUM in addressing these CSR issues. 

"Additionality refers to outputs that would not have been achieved in the market without the 
programme" (ToR). The ToR-document specifically mention the possibility that activities of PUM 
compete with (local) private consultancies, but there are more conditions for determining if a project 
can be qualified as additional or not. For the present evaluation we suggest using the "DCED model" 
to assess additionality, e.g. by checking if PUM applies the criteria listed in this modeP^. 


See for example the Policy letter 'Ondernemen voor ontwikkeling: investeren in duurzame en inclusieve groei' 30-09- 
2013 

"Demonstrating additionality in private sector development initiatives" (DCED, 2014) 
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Relevante and additionality will be assessed ex-ante and ex-post. In the ex-ante part of the analysis 
we focus on the procedures and criteria applied during the appraisal stage of the various projects 
(e.g. missions and sector-specific programmes). The evaluation will analyse the selection process of 
countries and sectors, the distribution of activities among selected countries and sectors and the 
determinants for approval or rejection of proposals. For the ex-post part of the analysis we assess to 
what extent the activities were actually relevant and additional and we look at the M&E Systems 
used to evaluate the results of the programme and its actual contribution to the Ministry's 
objectives, taking into consideration possible changes in policy objectives between 2012 and 2015 
and how these changes affected PUM. Here as well information gathered during the field missions 
will be used to conclude about ex post relevance and additionality. 

In total there are 7 EQs on relevance and additionality. It is important to onderstand the relations 
between the EQs. In fact, we observe three types of EQs under this heading, which helped us with 
identifying the type of indicators needed and the required sources of information. 


1. EQs about the procedures and criteria used to assess the relevance and additionality (EQl, 
EQ2 and EQ5). 

2. EQs about the quantity and quality of data and information available to assess the 
relevance and additionality (EQ3 and EQ5) 

3. EQs about the relevance and additionality of the programme (EQ4, EQ6 and EQ7) 

Effectiveness 

When applied to PUM, effectiveness measures "the extent to which an aid activity attains its direct 
objectives". The ToR state that "PUM projects are effective as far as they contribute to increased 
capacity and performance of the companies that benefitted from the activities." 

The effectiveness of PUM will be assessed using the Theory of Change and the results chain 
introduced above. While the concept of relevance focuses on longer-term results of the programme 
(including its impact on job creation and sustainable and inclusive growth), effectiveness clearly 
concentrates on the (immediate, intermediate and ultimate) outcomes that can be attributed to 
PUM, including aid and trade (e.g. through business links). The phrase "increased capacity and 
performance of companies" refers to changes in the knowledge, practices and performance of SM Es. 
Effectiveness will be analysed at the level of the programme and also at the level of individual 
projects. 

The aim of the present evaluation is not only to measure the programme's effectiveness, but also to 
analyse the underlying factors that explain effectiveness, such as: 

• Sector-specific conditions 

• Country-specific conditions (context) 

• Beneficiary-specific conditions 

• The acquisition, approval and M&E procedures and the role of the different players in these 
procedures (e.g. experts, country coördinators, sector coördinators, local representatives, 
entrepreneurs/directors) 

Moreover, the current evaluation reviews how PUM tries to (or could try to) optimize these 
conditions, e.g. by selecting other sectors or countries (changing the sectoral or geographic spread), 
changing the portfolio of activities (individual interventions versus sector-specific programmes) and 
through the so-called "demand system": the process of allocating an expert to a specific mission and 
monitoring the quality of experts. 
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The evaluation matrix includes 8 EQs on effectiveness. We distinguish two types of EQs with different 
types of indicators and different sources needed. 

1. EQs about the quality of monitoring the effectiveness, including the quality of data available 
(EQ8, EQIO, EQ14). 

2. EQs about the effectiveness of the programme and the conditions that influence 
effectiveness (EQ95^ EQll, EQ12, EQ13, EQ14, EQ15) 

Efficiency 

In general, efficiency is a measure that indicates "how economically resources/inputs (funds, 
expertise, time, etc.) are converted to results" (ToR). When applied to PUM, it is about using "the 
least costly resources possible in order to achieve the desired outcomes" (ToR). 

In our view, efficiency has to be assessed by comparing the inputs of the programme with its outputs, 
not with its outcomes as the second citation from the ToR suggests. The question is essentially how 
economically funds and time of staff and volunteers are converted into missions, trainings, projects, 
business links, etc. The relation between inputs and outcomes is captured by the concept of cost- 
effectiveness. 

In the present evaluation, efficiency will mainly be assessed on the level of the programme, as 
recommended by the ToR. Following the EQs for this evaluation criterion, however, we also look at 
conditions that explain efficiency such as the sectoral and geographic spread and the composition of 
the portfolio (sector-specific versus individual projects). This implies that efficiency will have to be 
measured at sub-programme levels such as sector, country and type of intervention. 

As stated above, efficiency and cost-effectiveness can be expressed as a ratio between inputs and 
outputs or outcomes. However, we must be aware that such an indicator is never sufficiënt to fully 
answer EQs about efficiency and cost-effectiveness. In the evaluation we will use three methods to 
assess efficiency: 

1. Comparing the efficiency and cost-effectiveness with other PSD programmes (benchmarks). 
To this end, we will use a study of Carnegie that compares overhead costs of various PSD 
programmes. 

2. Measuring changes in efficiency and cost-effectiveness over time: has PUM been able to 
improve its efficiency, for example in comparison with the situation before 2012? 

3. By reviewing if bureaucracy and (overhead) costs could have been avoided. A delay in the 
approval or implementation phase can be a signal of low efficiency only if such a delay could 
have been avoided without compromising the effectiveness and relevance of the 
intervention. 

The evaluation includes 4 EQs on efficiency and cost-effectiveness that could be labelled as follows: 

1. EQs about the efficiency of the programme (EQ16, EQ17) 

2. Qne EQ about the cost-effectiveness of the programme (EQ18) 

3. Qne EQ about the quantity and quality of data and Information available to assess efficiency 
(EQ16) 


It could be argued that EQ9 contains an element of "quality of monitoring" and "effectiveness". In the evaluation matrix, 
however, we focus on effectiveness, since the quality of monitoring is already addressed by EQ8. 
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4. One EQ about the cost-effectiveness of PRIME (EQ19) 

Forward looking questions 

In addition to the questions on relevante, additionality, effectiveness and efficiency, the present 
evaluation also aims to answer four forward-looking questions. 

The first forward looking question (EQ20) is about the relevante of PUM. It is stated that PUM is 
primarily an "aid" instrument. It is expected that PUM is currently more relevant for the Ministry's 
objectives to eradicate poverty and promote sustainable, inclusive growth than for the objective to 
promote trade and investment, and hence economie growth and development in The Netherlands. 
Assuming that this is indeed correct, the ToR formulate the following question: "Is there scope to 
strengthen the links between 'aid and trade' in the PUM program, while maintaining a development 
focus?". 

How to interpret this question? In our view the development focus refers to the Ministry's objective 
to help the poorest and most vulnerable groups, which implies - as we stated above - that the 
programme should keep its focus on the least developed and fragile countries and should bring 
benefits to the lowest income groups in all countries it serves So, what can be done to this end? In 
our view, there are three options to explore: 

1. Changes in the sectoral spread. Possibly we can demonstrate that the sector is an important 
predictor of combining success in aid and trade (see EQ4 and EQ7). 

2. Changes in the geographic spread without reducing the share of LDCs and fragile countries 
(see EQ4 and EQ7). 

3. Changes in the selection criteria and procedures. Such changes could assist PUM with 
selecting beneficiaries, experts and other volunteers with a higher chance of combining 
success in aid and trade (see EQl-3). 

Other EQs provide input for answering this EQ. However, it also requires to discuss the feasibility of 
changes with relevant stakeholders such as the Ministry of Foreign Affairs, the Ministry of Economie 
Affairs, VNO-NCW, and last but not least the PUM staff and volunteers (experts, country 
coördinators, sector coördinators). 

The second forward looking question (EQ21) asks if it is "possible to interest Dutch business more in 
contributing to PUM". In our view the private sector may contribute in various ways: 

1. Financially. By providing funds to PUM, for example as investment in trade with SMEs in 
developing countries. 

2. Through In-kind contributions, e.g. by supplying expertise/man-hours. 

The feasibility of such contributions will be discussed with the PUM staff and volunteers, but also 
with Dutch companies that participated in business links (PUM survey), the Ministry of Foreign 
Affairs, the Ministry of Economie Affairs and of course with VNO-NCW as representative of all 
employers in the Netherlands. 

The third forward looking question (EQ22) is about cost-effectiveness. The question results from 
lOB's conclusion that PSD programmes are "often implemented in isolation, while recipients often 
face a broader set of challenges". The present evaluation will therefore answer the following 
question "Would a closer collaboration of PUM with other PSD programmes lead to a more cost- 
effective approach?". This question clearly builds on the answer to EQ13 which reviews the 
possibilities to enhance effectiveness through such collaborations. EQ22 adds the elements of costs, 
for example of coordination between programmes. Through interviews with relevant stakeholders 
(see EQ20) we will gain insight in the cost-effectiveness of collaborative programmes in comparison 
with isolated programmes. 
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The fourth forward looking question (EQ23) follows up on all questions about the quantity and 
quality of Information and data collected by PUM and PRIME (EQ3,EQ8 and EQ16): "Is it possible to 
make better use of Information, generated by PUM, for helping to strengthen the business climate in 
countries where PUM is active?". The underlying rationale is that if Information can be used to this 
end, it would increase the relevance of PUM and possibly also the cost-effectiveness of PRIME. The 
term "business climate" can be linked to the country-specific conditions (context) that have been 
identified as possible predictors of effectiveness. So, the question is if the M&E system of PUM 
provides relevant insight in the role of such contextual factors in explaining the success of 
interventions and if this insight would help developing countries to improve their business climate. 
To answer this question, we will first have to review the quality of Information and data collected, 
then have interviews with PUM and PRIME, and finally have interviews/focus groups with volunteers 
(experts, sector coördinators, country coördinators) and beneficiaries (notably BSOs) and other 
stakeholders abroad. 

Specific indicators 

The evaluation matrix presents rather general and mostly qualitative indicators of efficiency, 
additionality, effectiveness and efficiency. This has been done for two reasons: 

1. Part of the evaluation is to identify and assess indicators made available by the existing M&E 
of PUM. 

2. A large part of the evaluation takes a more qualitative approach, using interviews and focus 
groups to collect in-depth Information; qualitative indicators are more in line with this 
approach. 
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Annex 3: Persons Interviewed 


Mr. 

Wisdom 

Abodakpi 

PUM Local representative Ghana 

Mr. 

Solomon John 

Alanade 

PUM Local representative Ghana 

Mr. 

Esteban 

Alvarez 

PUM Local Representative in Manizales 

Mr. 

William 

Amofe 

PUM Local representative Ghana 

Mr. 

Samuel Kwasi 

Ansu 

PUM Local representative Ghana 

Mrs. 

Martha Lucia 

Arévalo 

Royal Netherlands Embassy, Economie Affairs & 
Development Cooperation 

Mr. 

Shola 

Arsyad 

PUM Local representative Palembang 

Mrs. 

Margaret 

Asare 

PUM Local representative Ghana 

Mrs. 

Linda 

Beukers 

Holland House in Colombia 

Mr. 

Hans 

Beving 

PUM Sector coördinator 

Mr. 

Hans de 

Boer 

Chairman VNO-NCW/MKB 

Mr. 

Arie de 

Bondt 

PUM Sector coördinator 

Mr. 

Martin 

Brouwer 

PUM Country coördinator 

Mr. 

Henk 

Buddingh 

PUM expert 

Mr. 

Juan Alfonso 

Contreras 

PUM Local Representative in Bogota 

Mr. 

George 

Coster 

Country coördinator Indonesia (Java and Kalimantan) 

Mr. 

Jan van den 

Crommenacker 

PUM Sector coördinator 

Mr. 

Paul 

Dielissen 

PUM Sector coördinator 

Mrs. 

René van der 

Eeze 

PUM Country coördinator 

Mr. 

Ger van der 

Eijk 

PUM Sector coördinator 

Mr. 

Mr. 

Pedro 

Eijkelenboom 

Erwanto 

PUM Business Development Department 

PUM Local representative Lampung 

Mr. 

Frans 

Friedemann 

PUM Country coördinator 

Mr. 

Rens van 

Gelder 

PUM HR 

Mr. 

Fred 

Gobah-Tengey 

PUM Local representative Ghana 

Mr. 

Fred 

Graalman 

PUM Local Representative in Medellin 

Mrs. 

Dezentje 

Hamming 

FME-CWM, Chairperson 

Mr. 

Job 

Harms 

PRIME, Erasmus University Rotterdam 

Mr. 

Huub 

Heier 

PUM expert 

Mr. 

Leon 

Huson 

PUM Country Manager And Program Coördinator India 

Mr. 

Agung 

Irianto 

PUM Local representative Jakarta/Bandung 

Mr. 

Matthijs 

Jansen 

PUM Sector coördinator 

Mr. 

Pieter 

Jansen 

PUM country coördinator 

Mrs. 

Karin 

Jensma 

PUM Business Development 

Mr. 

Joris 

Jurriëns 

Royal Netherlands Embassy, Embassy Councillar in 
Bogota 

Mr. 

Arend 

Knol 

PUM Sector coördinator 

Mr. 

Arend 

Koekkoek 

PUM experts 

Mr. 

Johan 

Koeslag 

PUM Staff volunteer 

Mrs. 

Annelies 

Kuijpers 

PUM Sector coördinator 

Mr. 

Paul 

Kwose 

PUM Local representative Ghana 

Mrs. 

Ingrid van 

Leeuwen 

PUM project officer Indonesia 

Mr. 

Kees 

Lindenburg 

PUM country coördinator for Colombia 

Mr. 

Hans 

Luursema 

PUM CFO 

Mrs. 

Karen 

Maas 

PRIME, Erasmus University Rotterdam 
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Mr. 

Roland 

Martin 

Ministry of Foreign Affairs, BEB 

Mr. 

Alex 

Meerkerk 

PUM Head of Policy & Monitoring 

Mr. 

Rob 

Munter 

PUM expert 

Mr. 

Arend Jan 

Nel 

PUM Sector coördinator 

Mr. 

Cees 

Nieuwenhuizen 

PUM Business link coördinator 

Mr. 

Maarten van 

Noort 

PUM Country coördinator 

Mr. 

Haki 

Pamuk 

PRIME LEI Wageningen University & Research 

Mr. 

Danny 

Perenkuan 

PUM (former) Local representative Papua 

Mrs. 

Maribel Monge 

Pérez 

PUM Assistant of Country Coördinator for Colombia 

Mr. 

Leander 

Petit 

PUM Country coördinator 

Mr. 

Thijs van 

Praag 

PUM Director 

Mr. 

Arie 

Pronk 

PUM Country coördinator 

Mr. 

Peter 

Richtering 

Ministry of Foreign Affairs, DDE 

Mr. 

Henk 

Riemens 

PUM expert 

Mrs. 

Fedes van 

Rijn 

PRIME, LEI Wageningen University & Research 

Mr. 

Job 

Runhaar 

Ministry of Foreign Affairs, DDE 

Mr. 

John 

Schellen 

PUM Country coördinator 

Mr. 

John W. 

Serbrock 

PUM Country coördinator East Indonesia 

Mr. 

Cor 

Simans 

PUM Expert 

Mr. 

Harry 

Simorangkir 

PUM Local representative Manado 

Mr. 

Jan 

Sluis 

PUM Country coördinator 

Mr. 

René 

Sluis 

PUM Country coördinator 

Mr. 

Louis 

Smit 

PUM Sector coördinator 

Mr. 

Rolf 

Soedjak 

PUM Country coördinator Sumatra 

Mr. 

Aulia 

Sulaiman 

PUM Local representative Pekanbaru 

Mr. 

Harry 

Surminski 

PUM, Business link / sector coördinator 

Mr. 


Tarsini 

PUM Local representative Surabaya 

Mr. 

Fred 

Tassi 

PUM Sector coördinator 

Mr. 

Bouwe 

Taverne 

PUM Manager CSR 

Mr. 

Stevy 

Thioritz 

PUM Local representative Makassar 

Mr. 

Max 

Timmerman 

RVO/CBI 

Mr. 

Giel 

Ton 

PRIME, LEI Wageningen University & Research 

Mr. 

Adriana 

Uribe 

PUM Local Representative in Barranquilla 

Mr. 

Johan 

Veul 

Ministry of Foreign Affairs, DDE 

Mr. 

Eric 

Visser 

PUM expert 

Mr. 

Yuyun 

Yunastuti 

PUM Local representative Yogyakarta 

Mr. 

Rob 

Zaadnoordijk 

PUM Country coördinator 

Mr. 

Rinus van der 

Zedde 

PUM expert 

Mrs. 

Tess van der 

Zee 

Royal Netherlands Embassy Indonesia, Deputy Head & 


First Secretary Economie Section 
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